2024-04-12T15:48:00.330 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-12T15:48:00.330 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rzarzynski@teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} --name rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi --archive /home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654145 --verbose -- /home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654145/orig.config.yaml 2024-04-12T15:48:00.352 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T15:48:00.443 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654145 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} email: null first_in_suite: false job_id: '7654145' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d owner: scheduled_rzarzynski@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8148 sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d targets: smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHcJ/LglOklgXMH88njrkk534WEdOIsHeRT1ocPU7vO+2DB6/3zevQgvjesn20Y2jNx4T/3Bya+JOf8h0U52kA= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-12_15:15:07 tube: smithi user: rzarzynski verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-12T15:48:00.443 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa; will attempt to use it 2024-04-12T15:48:00.444 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks 2024-04-12T15:48:00.444 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-12T15:48:00.446 INFO:teuthology.task.internal:Checking packages... 2024-04-12T15:48:00.472 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'bf9c5618cc018927fa77780d25b6deb1f5dc254d' 2024-04-12T15:48:00.472 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T15:48:00.472 INFO:teuthology.packaging:ref: None 2024-04-12T15:48:00.472 INFO:teuthology.packaging:tag: None 2024-04-12T15:48:00.472 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-03-24-1159 2024-04-12T15:48:00.472 INFO:teuthology.packaging:sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d 2024-04-12T15:48:00.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri10-testing-2024-03-24-1159 2024-04-12T15:48:00.613 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2847-gbf9c5618-1jammy 2024-04-12T15:48:00.615 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-12T15:48:00.626 INFO:teuthology.task.internal:no buildpackages task found 2024-04-12T15:48:00.626 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-12T15:48:00.645 INFO:teuthology.task.internal:Saving configuration 2024-04-12T15:48:00.655 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-12T15:48:00.666 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T15:48:00.692 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654145', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 15:42:36.932597', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJeKWaLuEt/6AeyZRQLfio+RGh+eT3rYA+Sz1Hq4jr8o'} 2024-04-12T15:48:00.692 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-12T15:48:00.698 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-12T15:48:00.698 INFO:teuthology.run_tasks:Running task console_log... 2024-04-12T15:48:00.742 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0bf30c91f0>, signals=[15]) 2024-04-12T15:48:00.743 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-12T15:48:00.748 INFO:teuthology.task.internal:Opening connections... 2024-04-12T15:48:00.748 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-04-12T15:48:00.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T15:48:00.828 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-12T15:48:00.837 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-04-12T15:48:00.947 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-04-12T15:48:00.948 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-04-12T15:48:00.995 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-12T15:48:00.995 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2024-04-12T15:48:00.995 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="22.04" 2024-04-12T15:48:00.995 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-12T15:48:00.995 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=jammy 2024-04-12T15:48:00.995 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2024-04-12T15:48:00.995 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2024-04-12T15:48:00.996 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-12T15:48:00.996 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-12T15:48:00.996 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-12T15:48:00.996 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-12T15:48:00.996 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=jammy 2024-04-12T15:48:00.996 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-04-12T15:48:01.021 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-12T15:48:01.032 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-12T15:48:01.040 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-12T15:48:01.040 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-04-12T15:48:01.045 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-12T15:48:01.051 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-12T15:48:01.052 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-04-12T15:48:01.101 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-12T15:48:01.136 INFO:teuthology.run_tasks:Running task kernel... 2024-04-12T15:48:01.153 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-12T15:48:01.154 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-12T15:48:01.154 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T15:48:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T15:48:01.159 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-04-12T15:48:01.207 INFO:teuthology.orchestra.run.smithi137.stdout:5.15.0-56-generic 2024-04-12T15:48:01.207 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.15.0-56-generic 2024-04-12T15:48:01.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2024-04-12T15:48:01.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-04-12T15:48:01.477 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-12T15:48:01.589 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-12T15:48:01.670 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-12T15:48:01.696 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,343 kB] 2024-04-12T15:48:01.824 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [444 kB] 2024-04-12T15:48:01.829 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [237 kB] 2024-04-12T15:48:01.833 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-12T15:48:01.833 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,670 kB] 2024-04-12T15:48:01.858 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [34.1 kB] 2024-04-12T15:48:01.858 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [280 kB] 2024-04-12T15:48:01.861 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-12T15:48:01.861 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-12T15:48:01.869 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-12T15:48:01.875 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-12T15:48:01.876 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-12T15:48:01.877 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-12T15:48:01.877 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-12T15:48:01.898 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-12T15:48:01.917 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-12T15:48:02.023 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-12T15:48:02.162 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [610 kB] 2024-04-12T15:48:02.360 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,562 kB] 2024-04-12T15:48:02.510 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [297 kB] 2024-04-12T15:48:02.526 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-12T15:48:02.526 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.7 kB] 2024-04-12T15:48:02.526 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,702 kB] 2024-04-12T15:48:02.585 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [286 kB] 2024-04-12T15:48:02.593 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-12T15:48:02.757 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-12T15:48:03.343 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,074 kB] 2024-04-12T15:48:03.454 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-04-12T15:48:03.474 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-12T15:48:03.476 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-12T15:48:03.476 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.7 kB] 2024-04-12T15:48:03.480 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.1 kB] 2024-04-12T15:48:03.480 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-12T15:48:03.481 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-12T15:48:03.485 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-12T15:48:03.489 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-12T15:48:03.654 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-12T15:48:03.737 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-12T15:48:03.821 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-12T15:48:03.902 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-12T15:48:03.938 INFO:teuthology.orchestra.run.smithi137.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-12T15:48:08.248 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 12.9 MB in 3s (5,075 kB/s) 2024-04-12T15:48:09.507 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-04-12T15:48:09.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-12T15:48:09.585 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-04-12T15:48:09.776 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-04-12T15:48:09.777 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-04-12T15:48:09.930 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-04-12T15:48:09.930 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-12T15:48:09.930 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-12T15:48:09.931 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-12T15:48:09.931 INFO:teuthology.orchestra.run.smithi137.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-12T15:48:09.931 INFO:teuthology.orchestra.run.smithi137.stdout: python2.7-minimal 2024-04-12T15:48:09.931 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-12T15:48:09.932 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2024-04-12T15:48:09.932 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-12T15:48:09.932 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-generic linux-image-5.15.0-102-generic 2024-04-12T15:48:09.932 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-5.15.0-102-generic linux-modules-extra-5.15.0-102-generic 2024-04-12T15:48:09.933 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2024-04-12T15:48:09.934 INFO:teuthology.orchestra.run.smithi137.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-12T15:48:09.934 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2024-04-12T15:48:09.934 INFO:teuthology.orchestra.run.smithi137.stdout: thermald 2024-04-12T15:48:09.987 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2024-04-12T15:48:09.987 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-12T15:48:09.987 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.15.0-102-generic linux-modules-5.15.0-102-generic 2024-04-12T15:48:09.987 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-extra-5.15.0-102-generic 2024-04-12T15:48:09.988 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2024-04-12T15:48:09.989 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-12T15:48:10.167 INFO:teuthology.orchestra.run.smithi137.stdout:3 upgraded, 5 newly installed, 0 to remove and 313 not upgraded. 2024-04-12T15:48:10.167 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 113 MB of archives. 2024-04-12T15:48:10.167 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-12T15:48:10.167 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-102-generic amd64 5.15.0-102.112 [22.7 MB] 2024-04-12T15:48:11.672 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-102-generic amd64 5.15.0-102.112 [11.5 MB] 2024-04-12T15:48:12.000 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-102-generic amd64 5.15.0-102.112 [63.9 MB] 2024-04-12T15:48:14.174 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.102.99 [1,700 B] 2024-04-12T15:48:14.174 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.102.99 [2,512 B] 2024-04-12T15:48:14.175 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102 all 5.15.0-102.112 [12.4 MB] 2024-04-12T15:48:15.027 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102-generic amd64 5.15.0-102.112 [2,877 kB] 2024-04-12T15:48:15.101 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.102.99 [2,360 B] 2024-04-12T15:48:15.502 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 113 MB in 5s (22.2 MB/s) 2024-04-12T15:48:15.650 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-5.15.0-102-generic. 2024-04-12T15:48:17.595 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-12T15:48:17.600 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../0-linux-modules-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-12T15:48:17.678 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:20.054 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-image-5.15.0-102-generic. 2024-04-12T15:48:20.076 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../1-linux-image-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-12T15:48:20.163 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:20.624 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-102-generic. 2024-04-12T15:48:20.644 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-12T15:48:20.682 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:26.948 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../3-linux-generic_5.15.0.102.99_amd64.deb ... 2024-04-12T15:48:27.058 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-12T15:48:27.365 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.102.99_amd64.deb ... 2024-04-12T15:48:27.515 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-12T15:48:27.823 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-5.15.0-102. 2024-04-12T15:48:27.845 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../5-linux-headers-5.15.0-102_5.15.0-102.112_all.deb ... 2024-04-12T15:48:27.889 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-12T15:48:33.456 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-5.15.0-102-generic. 2024-04-12T15:48:33.493 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../6-linux-headers-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-12T15:48:33.530 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:36.126 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.102.99_amd64.deb ... 2024-04-12T15:48:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-12T15:48:36.592 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-12T15:48:36.713 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:36.839 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-generic (5.15.0.102.99) ... 2024-04-12T15:48:37.726 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:39.036 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-12T15:48:39.036 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-12T15:48:39.036 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-102-generic 2024-04-12T15:48:39.036 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-102-generic 2024-04-12T15:48:39.178 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:40.588 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:41.955 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-generic (5.15.0.102.99) ... 2024-04-12T15:48:42.081 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-generic (5.15.0.102.99) ... 2024-04-12T15:48:42.207 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-12T15:48:42.265 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-12T15:48:42.265 INFO:teuthology.orchestra.run.smithi137.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-102-generic 2024-04-12T15:48:56.389 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-12T15:48:56.390 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub' 2024-04-12T15:48:56.397 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-12T15:48:56.412 INFO:teuthology.orchestra.run.smithi137.stdout:Generating grub configuration file ... 2024-04-12T15:48:56.758 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-12T15:48:56.786 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-12T15:48:57.126 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-12T15:48:57.132 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-12T15:48:57.198 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-12T15:48:57.204 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-12T15:48:57.370 INFO:teuthology.orchestra.run.smithi137.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-12T15:48:57.370 INFO:teuthology.orchestra.run.smithi137.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-12T15:48:57.370 INFO:teuthology.orchestra.run.smithi137.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-12T15:48:57.419 INFO:teuthology.orchestra.run.smithi137.stdout:done 2024-04-12T15:48:57.884 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2024-04-12T15:48:57.903 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2024-04-12T15:48:57.903 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 21 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.15.0.102.99 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2024-04-12T15:48:57.904 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2024-04-12T15:48:57.905 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2024-04-12T15:48:57.905 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2024-04-12T15:48:57.905 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-102-generic 2024-04-12T15:48:57.905 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-12T15:48:57.905 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-12T15:48:57.905 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-12T15:48:57.905 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-04-12T15:48:57.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2024-04-12T15:48:58.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-04-12T15:48:58.257 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-12T15:48:58.269 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-12T15:48:58.270 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-12T15:48:58.290 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-12T15:48:59.621 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-04-12T15:48:59.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-12T15:48:59.691 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-04-12T15:48:59.867 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-04-12T15:48:59.867 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-04-12T15:49:00.022 INFO:teuthology.orchestra.run.smithi137.stdout:linux-image-generic is already the newest version (5.15.0.102.99). 2024-04-12T15:49:00.022 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-04-12T15:49:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-12T15:49:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-12T15:49:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-12T15:49:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-12T15:49:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-12T15:49:00.023 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-12T15:49:00.062 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 0 newly installed, 0 to remove and 313 not upgraded. 2024-04-12T15:49:00.063 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2024-04-12T15:49:00.121 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2024-04-12T15:49:00.121 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2024-04-12T15:49:00.121 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2024-04-12T15:49:00.121 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2024-04-12T15:49:00.121 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 21 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.15.0.102.99 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2024-04-12T15:49:00.122 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2024-04-12T15:49:00.122 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-04-12T15:49:00.166 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.tihostAfQ9 2024-04-12T15:49:00.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tihostAfQ9 2024-04-12T15:49:00.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.tihostAfQ9 2024-04-12T15:49:00.343 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.tihostAfQ9 is 10KB 2024-04-12T15:49:00.355 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.tihostAfQ9 2024-04-12T15:49:00.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-12T15:49:00.414 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-12T15:49:00.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-12T15:49:00.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-12T15:49:00.480 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-102-generic 2024-04-12T15:49:00.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo update-grub 2024-04-12T15:49:01.655 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub' 2024-04-12T15:49:01.656 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-12T15:49:01.659 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-04-12T15:49:01.911 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-12T15:49:01.916 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-12T15:49:02.222 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-12T15:49:02.227 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-12T15:49:02.268 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-12T15:49:02.273 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-12T15:49:02.410 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-12T15:49:02.411 INFO:teuthology.orchestra.run.smithi137.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-12T15:49:02.411 INFO:teuthology.orchestra.run.smithi137.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-12T15:49:02.422 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-04-12T15:49:02.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-04-12T15:49:32.470 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-12T15:49:32.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-12T15:49:32.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T15:49:51.007 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-04-12T15:50:00.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-12T15:50:00.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T15:50:03.074 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-04-12T15:50:15.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-12T15:50:15.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T15:50:46.833 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-04-12T15:50:47.825 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-12T15:50:47.825 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-102-generic"... 2024-04-12T15:50:47.825 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-04-12T15:50:47.872 INFO:teuthology.orchestra.run.smithi137.stdout:5.15.0-102-generic 2024-04-12T15:50:47.872 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-102-generic vs 5.15.0-102-generic 2024-04-12T15:50:47.872 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-12T15:50:47.873 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-12T15:50:48.873 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-12T15:50:48.873 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T15:50:48.937 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-04-12T15:50:48.958 DEBUG:teuthology.parallel:result is None 2024-04-12T15:50:48.958 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-12T15:50:48.966 INFO:teuthology.task.internal:Creating test directory... 2024-04-12T15:50:48.966 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T15:50:48.973 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-12T15:50:48.980 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-12T15:50:48.986 INFO:teuthology.task.internal:Creating archive directory... 2024-04-12T15:50:48.986 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T15:50:49.033 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-12T15:50:49.042 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-12T15:50:49.042 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T15:50:49.109 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T15:50:49.117 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T15:50:49.119 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-12T15:50:49.126 INFO:teuthology.task.internal:Configuring sudo... 2024-04-12T15:50:49.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T15:50:49.176 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-12T15:50:49.185 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-12T15:50:49.185 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T15:50:49.224 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T15:50:49.272 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T15:50:49.320 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-12T15:50:49.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T15:50:49.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-04-12T15:50:49.468 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-12T15:50:49.477 INFO:teuthology.task.internal:Starting timer... 2024-04-12T15:50:49.477 INFO:teuthology.run_tasks:Running task pcp... 2024-04-12T15:50:49.495 INFO:teuthology.run_tasks:Running task selinux... 2024-04-12T15:50:49.510 DEBUG:teuthology.task.selinux:Excluding smithi137: OS 'ubuntu' does not support SELinux 2024-04-12T15:50:49.510 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-12T15:50:49.510 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-12T15:50:49.510 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-12T15:50:49.510 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-12T15:50:49.519 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-12T15:50:49.540 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-12T15:50:49.542 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-12T15:57:46.974 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-04-12T15:57:46.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-12T15:57:46.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T15:57:47.080 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-04-12T15:57:47.158 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-12T15:57:47.158 INFO:teuthology.run_tasks:Running task clock... 2024-04-12T15:57:47.169 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-12T15:57:47.169 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T15:57:47.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T15:57:47.226 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-12T15:57:47.227 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Command line: ntpd -gq 2024-04-12T15:57:47.227 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: ---------------------------------------------------- 2024-04-12T15:57:47.227 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: ntp-4 is maintained by Network Time Foundation, 2024-04-12T15:57:47.227 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-12T15:57:47.227 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: corporation. Support and training for ntp-4 are 2024-04-12T15:57:47.227 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: available at https://www.nwtime.org/support 2024-04-12T15:57:47.227 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: ---------------------------------------------------- 2024-04-12T15:57:47.227 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: proto: precision = 0.045 usec (-24) 2024-04-12T15:57:47.228 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: basedate set to 2022-02-04 2024-04-12T15:57:47.228 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: gps base set to 2022-02-06 (week 2196) 2024-04-12T15:57:47.228 INFO:teuthology.orchestra.run.smithi137.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-12T15:57:47.229 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-12T15:57:47.229 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: restrict ::: KOD does nothing without LIMITED. 2024-04-12T15:57:47.229 INFO:teuthology.orchestra.run.smithi137.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-12T15:57:47.230 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Listen and drop on 0 v6wildcard [::]:123 2024-04-12T15:57:47.230 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-12T15:57:47.230 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Listen normally on 2 lo 127.0.0.1:123 2024-04-12T15:57:47.230 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Listen normally on 3 enp3s0f1 172.21.15.137:123 2024-04-12T15:57:47.230 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Listen normally on 4 lo [::1]:123 2024-04-12T15:57:47.230 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8137%5]:123 2024-04-12T15:57:47.230 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:47 ntpd[17203]: Listening on routing socket on fd #22 for interface updates 2024-04-12T15:57:48.227 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:48.227 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:48.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:48 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:50 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:52.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:52.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:52.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:52.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:52.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:52.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:52.230 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:52 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:54.227 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240412: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240412: Permission denied 2024-04-12T15:57:54.228 INFO:teuthology.orchestra.run.smithi137.stdout:12 Apr 15:57:54 ntpd[17203]: ntpd: time slew +0.001272 s 2024-04-12T15:57:54.229 INFO:teuthology.orchestra.run.smithi137.stdout:ntpd: time slew +0.001272s 2024-04-12T15:57:54.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-12T15:57:54.229 INFO:teuthology.orchestra.run.smithi137.stderr:12 Apr 15:57:54 ntpd[17203]: can't open /var/log/ntpstats/loopstats.20240412: Permission denied 2024-04-12T15:57:54.278 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2024-04-12T15:57:54.279 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2024-04-12T15:57:54.279 INFO:teuthology.orchestra.run.smithi137.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-12T15:57:54.279 INFO:teuthology.orchestra.run.smithi137.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-12T15:57:54.279 INFO:teuthology.orchestra.run.smithi137.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-12T15:57:54.279 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-12T15:57:54.279 INFO:teuthology.run_tasks:Running task install... 2024-04-12T15:57:54.287 DEBUG:teuthology.task.install:project ceph 2024-04-12T15:57:54.288 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'}} 2024-04-12T15:57:54.288 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'} 2024-04-12T15:57:54.288 INFO:teuthology.task.install:Using flavor: default 2024-04-12T15:57:54.294 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-12T15:57:54.294 INFO:teuthology.task.install:extra packages: [] 2024-04-12T15:57:54.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-key list | grep Ceph 2024-04-12T15:57:54.380 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-12T15:57:54.407 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-12T15:57:54.407 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph.com (release key) 2024-04-12T15:57:54.407 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-12T15:57:54.407 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=bf9c5618cc018927fa77780d25b6deb1f5dc254d 2024-04-12T15:57:54.564 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default/ 2024-04-12T15:57:54.564 INFO:teuthology.task.install.deb:Package version is 19.0.0-2847-gbf9c5618-1jammy 2024-04-12T15:57:54.681 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-12T15:57:54.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-12T15:57:54.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-04-12T15:57:54.881 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-12T15:57:54.912 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy InRelease 2024-04-12T15:57:54.933 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-12T15:57:54.955 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-12T15:57:54.976 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-04-12T15:57:54.983 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-12T15:57:55.004 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-12T15:57:55.061 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-12T15:57:55.142 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-12T15:57:55.278 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 29.2 kB in 0s (71.8 kB/s) 2024-04-12T15:57:56.198 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-04-12T15:57:56.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2847-gbf9c5618-1jammy cephadm=19.0.0-2847-gbf9c5618-1jammy ceph-mds=19.0.0-2847-gbf9c5618-1jammy ceph-mgr=19.0.0-2847-gbf9c5618-1jammy ceph-common=19.0.0-2847-gbf9c5618-1jammy ceph-fuse=19.0.0-2847-gbf9c5618-1jammy ceph-test=19.0.0-2847-gbf9c5618-1jammy ceph-volume=19.0.0-2847-gbf9c5618-1jammy radosgw=19.0.0-2847-gbf9c5618-1jammy python3-rados=19.0.0-2847-gbf9c5618-1jammy python3-rgw=19.0.0-2847-gbf9c5618-1jammy python3-cephfs=19.0.0-2847-gbf9c5618-1jammy python3-rbd=19.0.0-2847-gbf9c5618-1jammy libcephfs2=19.0.0-2847-gbf9c5618-1jammy libcephfs-dev=19.0.0-2847-gbf9c5618-1jammy librados2=19.0.0-2847-gbf9c5618-1jammy librbd1=19.0.0-2847-gbf9c5618-1jammy rbd-fuse=19.0.0-2847-gbf9c5618-1jammy 2024-04-12T15:57:56.276 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-04-12T15:57:56.470 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-04-12T15:57:56.471 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-04-12T15:57:56.645 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-04-12T15:57:56.646 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-12T15:57:56.646 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-12T15:57:56.646 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-12T15:57:56.646 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-12T15:57:56.647 INFO:teuthology.orchestra.run.smithi137.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-12T15:57:56.647 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-12T15:57:56.647 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2024-04-12T15:57:56.647 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-12T15:57:56.647 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-12T15:57:56.647 INFO:teuthology.orchestra.run.smithi137.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-12T15:57:56.647 INFO:teuthology.orchestra.run.smithi137.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-12T15:57:56.648 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-12T15:57:56.648 INFO:teuthology.orchestra.run.smithi137.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-12T15:57:56.648 INFO:teuthology.orchestra.run.smithi137.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-12T15:57:56.649 INFO:teuthology.orchestra.run.smithi137.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet zip 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-12T15:57:56.650 INFO:teuthology.orchestra.run.smithi137.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-12T15:57:56.651 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2024-04-12T15:57:56.651 INFO:teuthology.orchestra.run.smithi137.stdout: btrfs-tools 2024-04-12T15:57:56.747 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2024-04-12T15:57:56.748 INFO:teuthology.orchestra.run.smithi137.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-12T15:57:56.748 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-12T15:57:56.748 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-12T15:57:56.748 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-12T15:57:56.748 INFO:teuthology.orchestra.run.smithi137.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-12T15:57:56.748 INFO:teuthology.orchestra.run.smithi137.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-12T15:57:56.749 INFO:teuthology.orchestra.run.smithi137.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-12T15:57:56.749 INFO:teuthology.orchestra.run.smithi137.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-12T15:57:56.749 INFO:teuthology.orchestra.run.smithi137.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-12T15:57:56.750 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-12T15:57:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-12T15:57:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-12T15:57:56.751 INFO:teuthology.orchestra.run.smithi137.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-12T15:57:56.938 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 118 newly installed, 0 to remove and 300 not upgraded. 2024-04-12T15:57:56.938 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 180 MB of archives. 2024-04-12T15:57:56.938 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 823 MB of additional disk space will be used. 2024-04-12T15:57:56.938 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-12T15:57:56.995 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2847-gbf9c5618-1jammy [3,554 kB] 2024-04-12T15:57:57.369 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-12T15:57:57.379 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-12T15:57:57.445 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2847-gbf9c5618-1jammy [3,224 kB] 2024-04-12T15:57:57.463 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-12T15:57:57.643 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2847-gbf9c5618-1jammy [712 kB] 2024-04-12T15:57:57.682 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2847-gbf9c5618-1jammy [356 kB] 2024-04-12T15:57:57.696 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-12T15:57:57.707 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-12T15:57:57.709 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2847-gbf9c5618-1jammy [32.8 kB] 2024-04-12T15:57:57.710 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2847-gbf9c5618-1jammy [180 kB] 2024-04-12T15:57:57.717 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2847-gbf9c5618-1jammy [66.9 kB] 2024-04-12T15:57:57.721 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2847-gbf9c5618-1jammy [330 kB] 2024-04-12T15:57:57.737 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-12T15:57:57.747 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-12T15:57:57.747 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-12T15:57:57.749 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-12T15:57:57.749 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2847-gbf9c5618-1jammy [6,725 kB] 2024-04-12T15:57:57.767 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-12T15:57:57.936 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-12T15:57:58.054 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2847-gbf9c5618-1jammy [112 kB] 2024-04-12T15:57:58.056 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2847-gbf9c5618-1jammy [466 kB] 2024-04-12T15:57:58.075 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2847-gbf9c5618-1jammy [25.5 MB] 2024-04-12T15:57:58.338 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-12T15:57:58.372 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-12T15:57:58.374 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-12T15:57:58.454 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-12T15:57:58.505 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-12T15:57:58.527 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-12T15:57:58.534 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-12T15:57:58.534 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-12T15:57:58.546 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-12T15:57:58.546 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-12T15:57:58.711 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-12T15:57:58.794 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-12T15:57:58.823 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-12T15:57:58.840 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-12T15:57:58.884 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-12T15:57:59.007 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2847-gbf9c5618-1jammy [5,087 kB] 2024-04-12T15:57:59.088 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2847-gbf9c5618-1jammy [244 kB] 2024-04-12T15:57:59.093 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2847-gbf9c5618-1jammy [124 kB] 2024-04-12T15:57:59.097 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2847-gbf9c5618-1jammy [1,526 kB] 2024-04-12T15:57:59.098 INFO:teuthology.orchestra.run.smithi137.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-12T15:57:59.113 INFO:teuthology.orchestra.run.smithi137.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-12T15:57:59.120 INFO:teuthology.orchestra.run.smithi137.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-12T15:57:59.125 INFO:teuthology.orchestra.run.smithi137.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-12T15:57:59.147 INFO:teuthology.orchestra.run.smithi137.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-12T15:57:59.150 INFO:teuthology.orchestra.run.smithi137.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-12T15:57:59.152 INFO:teuthology.orchestra.run.smithi137.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2847-gbf9c5618-1jammy [6,258 kB] 2024-04-12T15:57:59.316 INFO:teuthology.orchestra.run.smithi137.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-12T15:57:59.364 INFO:teuthology.orchestra.run.smithi137.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2847-gbf9c5618-1jammy [22.8 MB] 2024-04-12T15:57:59.646 INFO:teuthology.orchestra.run.smithi137.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-12T15:57:59.677 INFO:teuthology.orchestra.run.smithi137.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-12T15:57:59.687 INFO:teuthology.orchestra.run.smithi137.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-12T15:57:59.890 INFO:teuthology.orchestra.run.smithi137.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-12T15:57:59.890 INFO:teuthology.orchestra.run.smithi137.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-12T15:57:59.897 INFO:teuthology.orchestra.run.smithi137.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-12T15:57:59.905 INFO:teuthology.orchestra.run.smithi137.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-12T15:57:59.918 INFO:teuthology.orchestra.run.smithi137.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-12T15:57:59.939 INFO:teuthology.orchestra.run.smithi137.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-12T15:57:59.945 INFO:teuthology.orchestra.run.smithi137.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-12T15:58:00.121 INFO:teuthology.orchestra.run.smithi137.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-12T15:58:00.205 INFO:teuthology.orchestra.run.smithi137.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2847-gbf9c5618-1jammy [14.1 kB] 2024-04-12T15:58:00.206 INFO:teuthology.orchestra.run.smithi137.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2847-gbf9c5618-1jammy [870 kB] 2024-04-12T15:58:00.223 INFO:teuthology.orchestra.run.smithi137.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2847-gbf9c5618-1jammy [2,435 kB] 2024-04-12T15:58:00.263 INFO:teuthology.orchestra.run.smithi137.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2847-gbf9c5618-1jammy [791 kB] 2024-04-12T15:58:00.273 INFO:teuthology.orchestra.run.smithi137.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2847-gbf9c5618-1jammy [148 kB] 2024-04-12T15:58:00.274 INFO:teuthology.orchestra.run.smithi137.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2847-gbf9c5618-1jammy [2,312 kB] 2024-04-12T15:58:00.343 INFO:teuthology.orchestra.run.smithi137.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2847-gbf9c5618-1jammy [8,625 kB] 2024-04-12T15:58:00.385 INFO:teuthology.orchestra.run.smithi137.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-12T15:58:00.563 INFO:teuthology.orchestra.run.smithi137.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-12T15:58:00.576 INFO:teuthology.orchestra.run.smithi137.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-12T15:58:00.596 INFO:teuthology.orchestra.run.smithi137.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-12T15:58:00.628 INFO:teuthology.orchestra.run.smithi137.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2847-gbf9c5618-1jammy [14.2 kB] 2024-04-12T15:58:00.628 INFO:teuthology.orchestra.run.smithi137.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2847-gbf9c5618-1jammy [51.0 MB] 2024-04-12T15:58:00.661 INFO:teuthology.orchestra.run.smithi137.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-12T15:58:00.959 INFO:teuthology.orchestra.run.smithi137.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-12T15:58:00.961 INFO:teuthology.orchestra.run.smithi137.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-12T15:58:00.964 INFO:teuthology.orchestra.run.smithi137.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-12T15:58:00.964 INFO:teuthology.orchestra.run.smithi137.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-12T15:58:00.967 INFO:teuthology.orchestra.run.smithi137.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-12T15:58:01.175 INFO:teuthology.orchestra.run.smithi137.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-12T15:58:01.612 INFO:teuthology.orchestra.run.smithi137.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-12T15:58:01.618 INFO:teuthology.orchestra.run.smithi137.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-12T15:58:01.909 INFO:teuthology.orchestra.run.smithi137.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-12T15:58:01.910 INFO:teuthology.orchestra.run.smithi137.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-12T15:58:01.911 INFO:teuthology.orchestra.run.smithi137.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-12T15:58:01.914 INFO:teuthology.orchestra.run.smithi137.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-12T15:58:01.915 INFO:teuthology.orchestra.run.smithi137.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-12T15:58:01.917 INFO:teuthology.orchestra.run.smithi137.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-12T15:58:01.918 INFO:teuthology.orchestra.run.smithi137.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-12T15:58:01.920 INFO:teuthology.orchestra.run.smithi137.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-12T15:58:02.102 INFO:teuthology.orchestra.run.smithi137.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-12T15:58:02.515 INFO:teuthology.orchestra.run.smithi137.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2847-gbf9c5618-1jammy [121 kB] 2024-04-12T15:58:02.517 INFO:teuthology.orchestra.run.smithi137.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2847-gbf9c5618-1jammy [39.0 kB] 2024-04-12T15:58:02.518 INFO:teuthology.orchestra.run.smithi137.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2847-gbf9c5618-1jammy [13.3 MB] 2024-04-12T15:58:02.545 INFO:teuthology.orchestra.run.smithi137.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-12T15:58:02.592 INFO:teuthology.orchestra.run.smithi137.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-12T15:58:02.627 INFO:teuthology.orchestra.run.smithi137.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-12T15:58:02.724 INFO:teuthology.orchestra.run.smithi137.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-12T15:58:02.764 INFO:teuthology.orchestra.run.smithi137.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-12T15:58:02.776 INFO:teuthology.orchestra.run.smithi137.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-12T15:58:02.782 INFO:teuthology.orchestra.run.smithi137.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-12T15:58:02.828 INFO:teuthology.orchestra.run.smithi137.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-12T15:58:02.831 INFO:teuthology.orchestra.run.smithi137.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-12T15:58:02.853 INFO:teuthology.orchestra.run.smithi137.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-12T15:58:02.943 INFO:teuthology.orchestra.run.smithi137.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/bf9c5618cc018927fa77780d25b6deb1f5dc254d/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2847-gbf9c5618-1jammy [91.4 kB] 2024-04-12T15:58:03.017 INFO:teuthology.orchestra.run.smithi137.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-12T15:58:03.290 INFO:teuthology.orchestra.run.smithi137.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-12T15:58:03.309 INFO:teuthology.orchestra.run.smithi137.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-12T15:58:03.463 INFO:teuthology.orchestra.run.smithi137.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-12T15:58:03.492 INFO:teuthology.orchestra.run.smithi137.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-12T15:58:03.649 INFO:teuthology.orchestra.run.smithi137.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-12T15:58:03.652 INFO:teuthology.orchestra.run.smithi137.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-12T15:58:03.653 INFO:teuthology.orchestra.run.smithi137.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-12T15:58:03.675 INFO:teuthology.orchestra.run.smithi137.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-12T15:58:03.681 INFO:teuthology.orchestra.run.smithi137.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-12T15:58:04.340 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 180 MB in 7s (25.6 MB/s) 2024-04-12T15:58:04.471 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-12T15:58:04.560 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-12T15:58:04.565 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-12T15:58:04.645 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-12T15:58:04.982 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-12T15:58:05.009 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-12T15:58:05.049 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-12T15:58:05.393 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-12T15:58:05.420 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-12T15:58:05.460 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-12T15:58:05.972 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-12T15:58:05.999 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-12T15:58:06.039 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-12T15:58:06.366 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-12T15:58:06.389 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-12T15:58:06.433 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-12T15:58:06.769 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-12T15:58:06.796 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-12T15:58:06.837 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-12T15:58:07.206 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-12T15:58:07.232 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-12T15:58:07.272 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-12T15:58:07.567 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librados2. 2024-04-12T15:58:07.593 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../007-librados2_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:07.625 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librados2 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:08.044 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librbd1. 2024-04-12T15:58:08.071 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../008-librbd1_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:08.111 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librbd1 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:08.547 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs2. 2024-04-12T15:58:08.573 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:08.614 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs2 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:08.891 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rados. 2024-04-12T15:58:08.906 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:08.949 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rados (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:09.244 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-12T15:58:09.270 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2847-gbf9c5618-1jammy_all.deb ... 2024-04-12T15:58:09.302 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-argparse (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:09.546 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cephfs. 2024-04-12T15:58:09.561 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:09.604 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cephfs (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:09.915 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-12T15:58:09.942 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2847-gbf9c5618-1jammy_all.deb ... 2024-04-12T15:58:09.982 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-common (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:10.310 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-12T15:58:10.337 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-12T15:58:10.377 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-12T15:58:10.913 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-prettytable. 2024-04-12T15:58:10.939 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-12T15:58:10.979 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-12T15:58:11.240 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rbd. 2024-04-12T15:58:11.266 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:11.307 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rbd (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:11.668 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-12T15:58:11.694 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-12T15:58:11.734 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-12T15:58:12.012 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-12T15:58:12.027 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-12T15:58:12.062 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-12T15:58:12.340 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-12T15:58:12.355 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-12T15:58:12.390 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-12T15:58:12.718 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua5.1. 2024-04-12T15:58:12.744 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-12T15:58:12.784 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-12T15:58:13.104 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-any. 2024-04-12T15:58:13.130 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-12T15:58:13.171 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-12T15:58:13.390 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package zip. 2024-04-12T15:58:13.405 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-12T15:58:13.440 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking zip (3.0-12build2) ... 2024-04-12T15:58:13.727 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package unzip. 2024-04-12T15:58:13.753 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-12T15:58:13.793 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-12T15:58:14.180 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package luarocks. 2024-04-12T15:58:14.206 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-12T15:58:14.247 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-12T15:58:14.621 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librgw2. 2024-04-12T15:58:14.647 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../025-librgw2_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:14.687 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librgw2 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:15.240 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rgw. 2024-04-12T15:58:15.255 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:15.298 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rgw (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:15.634 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-12T15:58:15.661 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-12T15:58:15.701 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-12T15:58:15.962 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libradosstriper1. 2024-04-12T15:58:15.988 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:16.029 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libradosstriper1 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:16.465 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-common. 2024-04-12T15:58:16.492 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:16.532 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-common (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:17.945 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-base. 2024-04-12T15:58:17.972 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:18.021 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-base (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:18.574 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-12T15:58:18.601 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-12T15:58:18.641 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-12T15:58:18.902 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cheroot. 2024-04-12T15:58:18.929 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-12T15:58:18.969 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-12T15:58:19.271 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-12T15:58:19.298 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-12T15:58:19.338 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-12T15:58:19.641 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-12T15:58:19.668 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-12T15:58:19.707 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-12T15:58:19.994 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-12T15:58:20.021 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-12T15:58:20.060 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-12T15:58:20.305 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tempora. 2024-04-12T15:58:20.331 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-12T15:58:20.371 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-12T15:58:20.674 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-portend. 2024-04-12T15:58:20.701 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-12T15:58:20.740 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-12T15:58:21.069 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-12T15:58:21.095 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-12T15:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-12T15:58:21.554 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-12T15:58:21.570 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-12T15:58:21.604 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-12T15:58:21.950 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-natsort. 2024-04-12T15:58:21.976 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-12T15:58:22.016 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-12T15:58:22.328 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-logutils. 2024-04-12T15:58:22.354 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-12T15:58:22.394 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-12T15:58:22.663 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-12T15:58:22.690 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-12T15:58:22.721 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-12T15:58:22.991 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-mako. 2024-04-12T15:58:23.017 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-12T15:58:23.057 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-12T15:58:23.403 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-12T15:58:23.429 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-12T15:58:23.469 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-12T15:58:23.755 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-12T15:58:23.782 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-12T15:58:23.822 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-12T15:58:24.083 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webob. 2024-04-12T15:58:24.109 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-12T15:58:24.149 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-12T15:58:24.502 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-waitress. 2024-04-12T15:58:24.529 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-12T15:58:24.572 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-12T15:58:24.846 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tempita. 2024-04-12T15:58:24.862 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-12T15:58:24.896 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-12T15:58:25.149 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-paste. 2024-04-12T15:58:25.165 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-12T15:58:25.207 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-12T15:58:25.561 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-12T15:58:25.576 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-12T15:58:25.619 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-12T15:58:25.922 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-12T15:58:25.948 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-12T15:58:25.988 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-12T15:58:26.266 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webtest. 2024-04-12T15:58:26.289 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-12T15:58:26.324 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-12T15:58:26.594 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pecan. 2024-04-12T15:58:26.620 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-12T15:58:26.652 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-12T15:58:26.974 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-certifi. 2024-04-12T15:58:27.000 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-12T15:58:27.031 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-12T15:58:27.326 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-idna. 2024-04-12T15:58:27.353 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-12T15:58:27.401 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-12T15:58:27.712 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-urllib3. 2024-04-12T15:58:27.739 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-12T15:58:27.779 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-12T15:58:28.099 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-requests. 2024-04-12T15:58:28.125 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-12T15:58:28.165 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-12T15:58:28.502 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-12T15:58:28.528 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-12T15:58:28.568 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-12T15:58:28.854 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-12T15:58:28.881 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2847-gbf9c5618-1jammy_all.deb ... 2024-04-12T15:58:28.921 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:29.234 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-12T15:58:29.249 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:29.283 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:29.595 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-12T15:58:29.622 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-12T15:58:29.669 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-12T15:58:29.972 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-openssl. 2024-04-12T15:58:29.999 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-12T15:58:30.039 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-12T15:58:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr. 2024-04-12T15:58:31.168 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:31.200 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:31.579 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mon. 2024-04-12T15:58:31.605 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:31.770 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mon (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:32.374 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-osd. 2024-04-12T15:58:32.400 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:32.440 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-osd (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:33.476 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph. 2024-04-12T15:58:33.492 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../066-ceph_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:33.526 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:33.796 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-fuse. 2024-04-12T15:58:33.822 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:33.862 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-fuse (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:34.249 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mds. 2024-04-12T15:58:34.275 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:34.315 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mds (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:34.735 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package cephadm. 2024-04-12T15:58:34.761 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../069-cephadm_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:34.801 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking cephadm (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:35.070 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-12T15:58:35.086 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-12T15:58:35.120 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-12T15:58:35.423 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-12T15:58:35.438 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-12T15:58:35.481 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-12T15:58:36.470 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-babel. 2024-04-12T15:58:36.497 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-12T15:58:36.536 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-12T15:58:36.847 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jinja2. 2024-04-12T15:58:36.874 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-12T15:58:36.914 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-12T15:58:37.233 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-12T15:58:37.260 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2847-gbf9c5618-1jammy_all.deb ... 2024-04-12T15:58:37.300 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:37.639 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-12T15:58:37.665 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-12T15:58:37.705 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-12T15:58:38.008 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-routes. 2024-04-12T15:58:38.034 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-12T15:58:38.074 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-12T15:58:38.411 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-12T15:58:38.437 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2847-gbf9c5618-1jammy_all.deb ... 2024-04-12T15:58:38.477 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:39.749 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-12T15:58:39.776 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-12T15:58:39.815 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-12T15:58:40.285 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-joblib. 2024-04-12T15:58:40.312 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-12T15:58:40.352 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-12T15:58:40.697 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-12T15:58:40.724 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-12T15:58:40.763 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-12T15:58:41.025 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn. 2024-04-12T15:58:41.051 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-12T15:58:41.091 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-12T15:58:41.741 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-12T15:58:41.769 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2847-gbf9c5618-1jammy_all.deb ... 2024-04-12T15:58:41.884 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:42.813 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cachetools. 2024-04-12T15:58:42.840 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-12T15:58:42.879 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-12T15:58:43.182 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-12T15:58:43.210 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-12T15:58:43.249 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-12T15:58:43.585 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-12T15:58:43.613 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-12T15:58:43.652 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-12T15:58:43.963 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rsa. 2024-04-12T15:58:43.991 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-12T15:58:44.030 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-12T15:58:44.334 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-google-auth. 2024-04-12T15:58:44.361 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-12T15:58:44.400 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-12T15:58:44.720 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-12T15:58:44.746 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-12T15:58:44.786 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-12T15:58:45.089 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-websocket. 2024-04-12T15:58:45.116 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-12T15:58:45.155 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-12T15:58:45.434 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-12T15:58:45.461 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-12T15:58:45.492 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-12T15:58:46.098 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-12T15:58:46.125 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2847-gbf9c5618-1jammy_all.deb ... 2024-04-12T15:58:46.164 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:46.483 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-12T15:58:46.511 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-12T15:58:46.550 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-12T15:58:46.887 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-12T15:58:46.914 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-12T15:58:47.003 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-12T15:58:47.397 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package jq. 2024-04-12T15:58:47.414 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-12T15:58:47.447 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-12T15:58:47.675 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package socat. 2024-04-12T15:58:47.692 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-12T15:58:47.733 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-12T15:58:48.028 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package xmlstarlet. 2024-04-12T15:58:48.055 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-12T15:58:48.094 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-12T15:58:48.449 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-test. 2024-04-12T15:58:48.476 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:48.515 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-test (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:50.761 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-volume. 2024-04-12T15:58:50.789 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2847-gbf9c5618-1jammy_all.deb ... 2024-04-12T15:58:50.827 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-volume (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:51.123 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-12T15:58:51.139 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:51.181 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs-dev (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:51.484 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-12T15:58:51.512 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-12T15:58:51.550 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-12T15:58:52.019 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-12T15:58:52.046 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-12T15:58:52.088 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-12T15:58:52.450 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package nvme-cli. 2024-04-12T15:58:52.477 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-12T15:58:52.516 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-12T15:58:52.947 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package pkg-config. 2024-04-12T15:58:52.974 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-12T15:58:53.013 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-12T15:58:53.274 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-12T15:58:53.302 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-12T15:58:53.341 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-12T15:58:53.769 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-12T15:58:53.797 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-12T15:58:53.835 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-12T15:58:54.122 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastescript. 2024-04-12T15:58:54.149 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-12T15:58:54.188 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-12T15:58:54.474 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pluggy. 2024-04-12T15:58:54.502 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-12T15:58:54.541 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-12T15:58:54.827 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-psutil. 2024-04-12T15:58:54.854 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-12T15:58:54.893 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-12T15:58:55.213 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-py. 2024-04-12T15:58:55.241 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-12T15:58:55.271 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-12T15:58:55.550 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pygments. 2024-04-12T15:58:55.577 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-12T15:58:55.616 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-12T15:58:56.078 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-12T15:58:56.106 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-12T15:58:56.144 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-12T15:58:56.456 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-toml. 2024-04-12T15:58:56.483 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-12T15:58:56.522 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-12T15:58:56.817 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pytest. 2024-04-12T15:58:56.844 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-12T15:58:56.883 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-12T15:58:57.212 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplejson. 2024-04-12T15:58:57.239 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-12T15:58:57.278 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-12T15:58:57.539 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-12T15:58:57.566 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-12T15:58:57.605 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-12T15:58:58.359 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package radosgw. 2024-04-12T15:58:58.387 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../116-radosgw_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:58.425 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking radosgw (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:59.195 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package rbd-fuse. 2024-04-12T15:58:59.211 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2847-gbf9c5618-1jammy_amd64.deb ... 2024-04-12T15:58:59.253 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking rbd-fuse (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:58:59.550 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-12T15:58:59.783 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-12T15:58:59.909 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-12T15:59:00.170 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-12T15:59:00.505 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-12T15:59:00.943 INFO:teuthology.orchestra.run.smithi137.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-12T15:59:01.011 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up cephadm (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:01.544 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user cephadm....done 2024-04-12T15:59:01.692 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-12T15:59:01.968 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-12T15:59:02.211 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-12T15:59:02.337 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-12T15:59:02.630 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-12T15:59:02.881 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-12T15:59:02.982 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-12T15:59:03.267 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-12T15:59:03.635 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-12T15:59:03.886 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-12T15:59:04.188 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-12T15:59:04.414 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-12T15:59:04.681 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-12T15:59:04.941 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-12T15:59:05.183 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-argparse (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:05.485 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-12T15:59:05.594 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-12T15:59:05.695 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-12T15:59:05.812 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-12T15:59:06.072 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-12T15:59:06.430 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-12T15:59:06.462 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-12T15:59:06.508 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-12T15:59:06.634 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-12T15:59:06.994 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-12T15:59:07.095 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-12T15:59:07.354 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-12T15:59:07.622 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-12T15:59:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up zip (3.0-12build2) ... 2024-04-12T15:59:08.016 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-12T15:59:08.701 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-12T15:59:08.944 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-12T15:59:09.070 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-12T15:59:09.196 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-12T15:59:09.489 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-12T15:59:09.873 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-12T15:59:10.099 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-12T15:59:10.459 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-12T15:59:10.752 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-12T15:59:11.095 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-12T15:59:11.355 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-12T15:59:11.480 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-common (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:11.782 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-idna (3.3-1) ... 2024-04-12T15:59:12.083 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-12T15:59:13.205 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-12T15:59:13.415 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-12T15:59:13.524 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-12T15:59:13.775 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-12T15:59:13.901 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-12T15:59:14.027 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-12T15:59:14.329 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-12T15:59:14.588 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-12T15:59:14.839 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-12T15:59:14.940 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-12T15:59:15.233 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-12T15:59:15.484 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-12T15:59:15.744 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-12T15:59:16.003 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-12T15:59:16.254 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-12T15:59:16.514 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-12T15:59:16.631 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-12T15:59:16.924 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-12T15:59:17.042 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-12T15:59:17.343 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-12T15:59:17.669 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-12T15:59:18.287 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-12T15:59:18.413 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-12T15:59:18.514 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-12T15:59:18.587 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-12T15:59:18.824 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-12T15:59:18.942 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-12T15:59:19.335 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-12T15:59:19.594 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-12T15:59:19.729 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-12T15:59:19.988 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-12T15:59:20.114 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-12T15:59:20.432 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-12T15:59:20.733 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-12T15:59:21.030 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-12T15:59:21.151 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-12T15:59:21.428 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-12T15:59:21.696 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-12T15:59:22.040 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-12T15:59:22.400 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-12T15:59:22.685 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-12T15:59:23.023 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-12T15:59:23.140 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librados2 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:23.241 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:23.367 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-12T15:59:23.703 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs2 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:23.829 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libradosstriper1 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:23.964 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-12T15:59:24.240 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librbd1 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:24.350 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-modules-core (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:24.476 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-fuse (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:24.698 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-12T15:59:24.698 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-12T15:59:25.202 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs-dev (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:25.328 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rados (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:25.454 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librgw2 (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:25.580 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-12T15:59:26.767 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rbd (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:26.893 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up rbd-fuse (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:27.019 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rgw (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:27.145 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cephfs (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:27.271 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-common (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:27.797 INFO:teuthology.orchestra.run.smithi137.stdout:Adding group ceph....done 2024-04-12T15:59:28.157 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user ceph....done 2024-04-12T15:59:28.308 INFO:teuthology.orchestra.run.smithi137.stdout:Setting system user ceph properties....done 2024-04-12T15:59:28.325 INFO:teuthology.orchestra.run.smithi137.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-12T15:59:28.462 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-12T15:59:28.805 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-12T15:59:29.321 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-test (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:29.431 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up radosgw (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:29.901 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-12T15:59:29.901 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-12T15:59:30.358 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-base (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:30.701 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-12T15:59:31.160 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mds (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:31.385 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-12T15:59:31.386 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-12T15:59:31.895 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:32.108 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-12T15:59:32.108 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-12T15:59:32.621 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-osd (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:32.907 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-12T15:59:32.907 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-12T15:59:33.422 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:33.532 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:33.709 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mon (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:33.922 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-12T15:59:33.922 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-12T15:59:34.443 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-cephadm (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:34.586 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:34.712 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-dashboard (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:34.879 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-volume (19.0.0-2847-gbf9c5618-1jammy) ... 2024-04-12T15:59:35.241 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-12T15:59:35.367 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-12T15:59:35.636 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-12T15:59:36.711 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-12T15:59:37.368 INFO:teuthology.orchestra.run.smithi137.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-12T15:59:37.374 DEBUG:teuthology.parallel:result is None 2024-04-12T15:59:37.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=bf9c5618cc018927fa77780d25b6deb1f5dc254d 2024-04-12T15:59:37.729 DEBUG:teuthology.orchestra.run.smithi137:> dpkg-query -W -f '${Version}' ceph 2024-04-12T15:59:37.749 INFO:teuthology.orchestra.run.smithi137.stdout:19.0.0-2847-gbf9c5618-1jammy 2024-04-12T15:59:37.749 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2847-gbf9c5618-1jammy 2024-04-12T15:59:37.749 INFO:teuthology.task.install:The correct ceph version 19.0.0-2847-gbf9c5618-1jammy is installed. 2024-04-12T15:59:37.751 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-12T15:59:37.751 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-12T15:59:37.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T15:59:37.800 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-12T15:59:37.801 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-12T15:59:37.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T15:59:37.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T15:59:37.909 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-12T15:59:37.909 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-12T15:59:37.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T15:59:37.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T15:59:38.017 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-12T15:59:38.017 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-12T15:59:38.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T15:59:38.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T15:59:38.129 INFO:teuthology.run_tasks:Running task workunit... 2024-04-12T15:59:38.146 INFO:tasks.workunit:Pulling workunits from ref bf9c5618cc018927fa77780d25b6deb1f5dc254d 2024-04-12T15:59:38.147 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-12T15:59:38.147 INFO:tasks.workunit:timeout=3h 2024-04-12T15:59:38.147 INFO:tasks.workunit:cleanup=True 2024-04-12T15:59:38.147 DEBUG:teuthology.orchestra.run.smithi137:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-12T15:59:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T15:59:38.178 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-12T15:59:38.178 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-12T15:59:38.226 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-12T15:59:38.226 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-12T15:59:38.274 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bf9c5618cc018927fa77780d25b6deb1f5dc254d 2024-04-12T15:59:38.324 INFO:tasks.workunit.client.0.smithi137.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-12T16:01:13.467 INFO:tasks.workunit.client.0.smithi137.stderr:Note: switching to 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'. 2024-04-12T16:01:13.468 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-04-12T16:01:13.468 INFO:tasks.workunit.client.0.smithi137.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-12T16:01:13.468 INFO:tasks.workunit.client.0.smithi137.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-12T16:01:13.468 INFO:tasks.workunit.client.0.smithi137.stderr:state without impacting any branches by switching back to a branch. 2024-04-12T16:01:13.468 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr: git switch -c 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr:Or undo this operation with: 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr: git switch - 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr: 2024-04-12T16:01:13.469 INFO:tasks.workunit.client.0.smithi137.stderr:HEAD is now at bf9c5618cc0 Merge branch 'wip-osd-ec-partial-reads' of https://github.com/rzarzynski/ceph into wip-yuri10-testing-2024-04-09-1003 2024-04-12T16:01:13.473 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-12T16:01:13.520 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-12T16:01:13.520 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-12T16:01:13.566 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-04-12T16:01:13.567 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-04-12T16:01:13.567 DEBUG:teuthology.orchestra.run.smithi137:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf9c5618cc018927fa77780d25b6deb1f5dc254d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ TIMEOUT=300 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ MAX_TIMEOUT=15 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ PG_NUM=4 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ TMPDIR=/tmp 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ type xmlstarlet 2024-04-12T16:01:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:++ XMLSTARLET=xmlstarlet 2024-04-12T16:01:13.622 INFO:tasks.workunit.client.0.smithi137.stderr:+++ uname 2024-04-12T16:01:13.622 INFO:tasks.workunit.client.0.smithi137.stderr:++ '[' Linux = FreeBSD ']' 2024-04-12T16:01:13.622 INFO:tasks.workunit.client.0.smithi137.stderr:++ SED=sed 2024-04-12T16:01:13.622 INFO:tasks.workunit.client.0.smithi137.stderr:++ AWK=awk 2024-04-12T16:01:13.623 INFO:tasks.workunit.client.0.smithi137.stderr:+++ stty -a 2024-04-12T16:01:13.623 INFO:tasks.workunit.client.0.smithi137.stderr:+++ head -1 2024-04-12T16:01:13.623 INFO:tasks.workunit.client.0.smithi137.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-04-12T16:01:13.635 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-12T16:01:13.635 INFO:tasks.workunit.client.0.smithi137.stderr:++ termwidth= 2024-04-12T16:01:13.636 INFO:tasks.workunit.client.0.smithi137.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-04-12T16:01:13.636 INFO:tasks.workunit.client.0.smithi137.stderr:++ DIFFCOLOPTS='-y ' 2024-04-12T16:01:13.636 INFO:tasks.workunit.client.0.smithi137.stderr:++ KERNCORE=kernel.core_pattern 2024-04-12T16:01:13.636 INFO:tasks.workunit.client.0.smithi137.stderr:++ EXTRA_OPTS= 2024-04-12T16:01:13.639 INFO:tasks.workunit.client.0.smithi137.stderr:++ test '' = TESTS 2024-04-12T16:01:13.639 INFO:tasks.workunit.client.0.smithi137.stderr:++ uname -m 2024-04-12T16:01:13.641 INFO:tasks.workunit.client.0.smithi137.stderr:+ arch=x86_64 2024-04-12T16:01:13.641 INFO:tasks.workunit.client.0.smithi137.stderr:+ case $arch in 2024-04-12T16:01:13.641 INFO:tasks.workunit.client.0.smithi137.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-04-12T16:01:13.641 INFO:tasks.workunit.client.0.smithi137.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-04-12T16:01:13.641 INFO:tasks.workunit.client.0.smithi137.stderr:+ plugins=(jerasure shec lrc isa) 2024-04-12T16:01:13.641 INFO:tasks.workunit.client.0.smithi137.stderr:+ main test-erasure-code-plugins 2024-04-12T16:01:13.641 INFO:tasks.workunit.client.0.smithi137.stderr:+ local dir=td/test-erasure-code-plugins 2024-04-12T16:01:13.641 INFO:tasks.workunit.client.0.smithi137.stderr:+ shift 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:+ shopt -s -o xtrace 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-code-plugins 2024-04-12T16:01:13.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-04-12T16:01:13.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-04-12T16:01:13.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-04-12T16:01:13.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-04-12T16:01:13.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-04-12T16:01:13.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-04-12T16:01:13.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none ' 2024-04-12T16:01:13.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-04-12T16:01:13.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-04-12T16:01:13.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-12T16:01:13.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-04-12T16:01:13.646 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_preload_no_warning 2024-04-12T16:01:13.646 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_preload_no_warning_default 2024-04-12T16:01:13.646 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_preload_warning' 2024-04-12T16:01:13.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-04-12T16:01:13.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-04-12T16:01:13.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-04-12T16:01:13.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:01:13.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:01:13.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:01:13.648 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:01:13.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:01:13.649 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:01:13.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:01:13.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:01:13.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:01:13.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:01:13.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:01:13.652 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:01:13.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:01:13.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:01:13.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:01:13.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:01:13.655 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:01:13.655 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:01:13.655 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:01:13.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:01:13.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:01:13.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:01:13.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:01:13.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:01:13.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:01:13.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:01:13.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:01:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-04-12T16:01:14.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:01:14.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:01:14.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:01:14.263 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:01:14.263 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:01:14.263 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:01:14.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:01:14.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:01:14.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:01:14.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:01:14.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:01:14.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:01:14.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:01:14.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:01:14.473 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:01:14.473 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:01:14.473 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:01:14.473 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:01:14.473 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:01:14.473 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:01:14.473 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:01:14.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:01:14.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:01:14.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:01:14.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:01:14.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:01:14.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:01:14.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:01:14.591 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:01:14.591 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:01:14.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:01:14.592 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:01:14.592 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:01:14.592 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:01:14.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:01:14.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:01:14.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:01:14.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:01:14.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:01:14.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:01:14.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:01:14.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:01:14.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:01:14.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:01:15.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:01:15.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:01:15.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:01:15.585 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:01:15.585 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:01:15.585 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:01:15.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:01:15.586 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:01:15.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:01:15.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:01:15.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:01:15.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:01:15.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:01:15.718 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:01:15.718 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 4d1a167e-e2ea-40f6-b99f-6c6367611117 2024-04-12T16:01:15.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d1a167e-e2ea-40f6-b99f-6c6367611117 2024-04-12T16:01:15.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4d1a167e-e2ea-40f6-b99f-6c6367611117' 2024-04-12T16:01:15.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:01:15.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLWhlmoiXTKxAA50DySpEbpWami9Y3Tl2Ccw== 2024-04-12T16:01:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLWhlmoiXTKxAA50DySpEbpWami9Y3Tl2Ccw=="}' 2024-04-12T16:01:15.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d1a167e-e2ea-40f6-b99f-6c6367611117 -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:01:16.008 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:01:16.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:01:16.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLWhlmoiXTKxAA50DySpEbpWami9Y3Tl2Ccw== --osd-uuid 4d1a167e-e2ea-40f6-b99f-6c6367611117 2024-04-12T16:01:16.099 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:16.098+0000 7f863d6d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:16.101 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:16.098+0000 7f863d6d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:16.103 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:16.102+0000 7f863d6d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:16.103 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:16.102+0000 7f863d6d07c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:01:16.103 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:16.102+0000 7f863d6d07c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:01:16.103 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:16.102+0000 7f863d6d07c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:01:16.159 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:16.158+0000 7f863d6d07c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:01:22.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:01:22.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:01:22.131 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:01:22.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:01:22.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:01:22.396 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:01:22.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:01:22.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:01:22.425 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:22.422+0000 7f7d4eef27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:22.428 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:22.426+0000 7f7d4eef27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:22.429 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:22.426+0000 7f7d4eef27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:22.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:01:22.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:01:22.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:01:22.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:01:22.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:01:22.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:22.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:01:22.613 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:01:22.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:22.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:22.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:22.972 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:22.970+0000 7f7d4eef27c0 -1 Falling back to public interface 2024-04-12T16:01:23.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:23.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:23.831 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:01:23.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:01:23.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:23.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:24.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:25.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:25.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:25.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:01:25.127 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:01:25.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:25.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:25.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:25.602 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:25.602+0000 7f7d4eef27c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:01:26.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:26.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:26.549 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:01:26.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:01:26.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:26.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:27.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:28.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:28.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:28.013 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:01:28.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:01:28.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:28.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:28.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:29.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:29.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:29.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:01:29.554 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:01:29.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:29.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:29.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:30.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:30.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:30.978 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:01:30.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:01:30.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:30.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:31.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:32.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:32.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:32.436 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:01:32.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:01:32.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:32.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:32.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:33.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:33.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:33.904 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:01:33.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:01:33.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:33.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:34.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:35.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:35.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:35.460 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:01:35.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:01:35.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:35.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:35.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:36.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:36.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:36.925 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:01:36.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:01:36.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:36.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:37.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:38.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:38.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:38.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:01:38.494 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:01:38.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:38.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:39.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:40.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:40.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:40.505 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:01:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:01:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:40.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:41.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:41.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:41.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:01:41.940 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:01:41.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:41.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:42.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:43.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:43.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:43.360 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:01:43.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:01:43.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:43.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:44.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:45.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:45.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:45.093 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:01:45.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:01:45.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:45.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:46.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:47.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:47.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:47.011 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:01:47.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:01:47.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:47.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:48.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:49.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:49.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:49.122 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:01:49.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:01:49.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:49.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:01:49.605 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1505948884,v1:127.0.0.1:6801/1505948884] [v2:127.0.0.1:6802/1505948884,v1:127.0.0.1:6803/1505948884] exists,up 4d1a167e-e2ea-40f6-b99f-6c6367611117 2024-04-12T16:01:49.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:01:49.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:01:49.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:01:49.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-04-12T16:01:49.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-04-12T16:01:49.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:01:49.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:01:49.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:01:49.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:01:49.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:01:49.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-04-12T16:01:49.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:01:49.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17f41d40-a533-4945-bae5-5f89e2b35fff 2024-04-12T16:01:49.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 17f41d40-a533-4945-bae5-5f89e2b35fff' 2024-04-12T16:01:49.613 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 17f41d40-a533-4945-bae5-5f89e2b35fff 2024-04-12T16:01:49.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:01:49.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtWhlmC9qtJRAAI1XPwmuKQkMHlnRn0RjAbg== 2024-04-12T16:01:49.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtWhlmC9qtJRAAI1XPwmuKQkMHlnRn0RjAbg=="}' 2024-04-12T16:01:49.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17f41d40-a533-4945-bae5-5f89e2b35fff -i td/test-erasure-code-plugins/1/new.json 2024-04-12T16:01:50.115 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:01:50.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-04-12T16:01:50.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtWhlmC9qtJRAAI1XPwmuKQkMHlnRn0RjAbg== --osd-uuid 17f41d40-a533-4945-bae5-5f89e2b35fff 2024-04-12T16:01:50.163 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:50.162+0000 7f801fa657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:50.165 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:50.166+0000 7f801fa657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:50.167 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:50.166+0000 7f801fa657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:50.168 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:50.166+0000 7f801fa657c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-04-12T16:01:50.168 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:50.166+0000 7f801fa657c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-04-12T16:01:50.168 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:50.166+0000 7f801fa657c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-04-12T16:01:50.249 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:50.250+0000 7f801fa657c0 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-04-12T16:01:56.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-04-12T16:01:56.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:01:56.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:01:56.122 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:01:56.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:01:56.673 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:01:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:01:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:01:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:01:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:01:56.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:01:56.704 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:56.702+0000 7fd93baa97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:56.722 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:56.722+0000 7fd93baa97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:56.724 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:56.722+0000 7fd93baa97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:01:57.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:01:57.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:01:57.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:01:57.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:01:57.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:01:57.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:57.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:01:57.119 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:01:57.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:57.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:01:57.290 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:57.291+0000 7fd93baa97c0 -1 Falling back to public interface 2024-04-12T16:01:57.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:58.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:58.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:58.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:01:58.543 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:01:58.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:58.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:01:58.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:01:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:01:59.831+0000 7fd93baa97c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:01:59.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:01:59.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:01:59.951 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:01:59.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:01:59.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:01:59.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:00.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:01.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:01.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:01.483 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:02:01.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:02:01.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:01.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:02.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:03.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:03.017 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:02:03.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:03.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:02:03.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:03.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:03.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:04.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:04.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:04.611 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:02:04.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:02:04.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:04.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:05.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:06.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:06.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:06.074 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:02:06.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:02:06.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:06.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:06.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:07.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:07.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:07.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:02:07.622 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:02:07.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:07.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:08.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:09.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:09.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:09.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:02:09.042 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:02:09.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:09.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:09.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:10.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:10.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:10.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:02:10.453 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:02:10.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:10.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:10.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:11.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:11.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:11.861 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:02:11.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:02:11.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:11.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:12.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:13.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:13.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:13.558 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:02:13.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:02:13.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:13.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:14.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:15.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:15.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:15.351 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:02:15.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:02:15.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:15.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:16.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:17.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:17.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:17.132 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:02:17.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:02:17.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:17.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:17.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:18.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:18.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:18.813 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:02:18.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:02:18.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:18.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:19.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:20.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:20.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:20.341 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:02:20.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:02:20.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:20.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:21.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:22.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:22.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:22.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:02:22.387 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:02:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:02:22.851 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1704894659,v1:127.0.0.1:6811/1704894659] [v2:127.0.0.1:6812/1704894659,v1:127.0.0.1:6813/1704894659] exists,up 17f41d40-a533-4945-bae5-5f89e2b35fff 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:02:22.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:02:22.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:02:22.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:02:22.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:02:22.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:02:22.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:02:22.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:02:22.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:02:22.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-04-12T16:02:22.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:02:22.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b62946b-03bc-428b-867c-c1fda711fe5a 2024-04-12T16:02:22.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4b62946b-03bc-428b-867c-c1fda711fe5a' 2024-04-12T16:02:22.861 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 4b62946b-03bc-428b-867c-c1fda711fe5a 2024-04-12T16:02:22.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:02:22.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOWxlm6WZwNBAAw5xgvWYnfdv6AcF4p1NCLg== 2024-04-12T16:02:22.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOWxlm6WZwNBAAw5xgvWYnfdv6AcF4p1NCLg=="}' 2024-04-12T16:02:22.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b62946b-03bc-428b-867c-c1fda711fe5a -i td/test-erasure-code-plugins/2/new.json 2024-04-12T16:02:23.352 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:02:23.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-04-12T16:02:23.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOWxlm6WZwNBAAw5xgvWYnfdv6AcF4p1NCLg== --osd-uuid 4b62946b-03bc-428b-867c-c1fda711fe5a 2024-04-12T16:02:23.393 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:23.395+0000 7f06389277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:02:23.395 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:23.395+0000 7f06389277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:02:23.397 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:23.395+0000 7f06389277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:02:23.397 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:23.399+0000 7f06389277c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-04-12T16:02:23.397 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:23.399+0000 7f06389277c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-04-12T16:02:23.397 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:23.399+0000 7f06389277c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-04-12T16:02:23.476 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:23.475+0000 7f06389277c0 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-04-12T16:02:29.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-04-12T16:02:29.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:02:29.263 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:02:29.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:02:29.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:02:29.811 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:02:29.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:02:29.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:02:29.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:02:29.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:02:29.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:02:29.835 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:29.835+0000 7f98ceb337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:02:29.848 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:29.847+0000 7f98ceb337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:02:29.849 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:29.851+0000 7f98ceb337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:02:30.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:02:30.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:02:30.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:02:30.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:02:30.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:02:30.261 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:02:30.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:30.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:02:30.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:30.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:30.422 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:30.423+0000 7f98ceb337c0 -1 Falling back to public interface 2024-04-12T16:02:30.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:31.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:31.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:31.653 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:02:31.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:02:31.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:31.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:32.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:33.032 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:02:33.031+0000 7f98ceb337c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:02:33.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:33.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:33.077 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:02:33.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:02:33.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:33.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:33.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:34.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:34.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:34.690 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:02:34.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:02:34.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:34.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:35.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:36.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:36.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:36.169 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:02:36.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:02:36.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:36.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:36.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:37.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:37.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:37.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:02:37.600 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:02:37.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:37.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:38.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:39.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:39.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:39.189 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:02:39.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:02:39.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:39.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:39.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:40.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:40.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:40.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:02:40.597 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:02:40.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:40.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:41.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:42.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:42.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:42.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:02:42.227 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:02:42.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:42.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:42.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:43.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:43.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:43.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:02:43.650 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:02:43.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:43.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:44.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:45.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:45.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:45.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:02:45.117 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:02:45.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:45.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:45.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:46.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:46.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:46.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:02:46.911 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:02:46.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:46.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:47.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:48.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:48.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:48.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:02:48.930 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:02:48.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:48.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:49.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:50.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:50.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:50.922 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:02:50.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:02:50.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:50.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:51.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:52.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:52.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:52.703 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:02:52.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:02:52.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:52.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:53.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:54.754 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:02:54.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:02:54.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:54.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:55.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:02:56.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:02:56.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:02:56.177 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:02:56.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:02:56.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:02:56.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:02:56.591 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1426184907,v1:127.0.0.1:6819/1426184907] [v2:127.0.0.1:6820/1426184907,v1:127.0.0.1:6821/1426184907] exists,up 4b62946b-03bc-428b-867c-c1fda711fe5a 2024-04-12T16:02:56.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:02:56.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:02:56.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:02:56.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-04-12T16:02:56.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-12T16:02:57.013 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-04-12T16:02:57.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-12T16:02:57.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:02:57.551 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:02:57.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:02:58.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-12T16:03:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-04-12T16:03:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:03:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:03:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:03:00.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:03:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:03:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:03:00.146 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:03:00.146 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:03:00.146 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:03:00.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:03:00.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:03:00.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:03:00.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:03:00.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:03:00.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:03:00.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:03:00.720 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:03:00.720 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T16:03:00.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:03:00.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:03:00.721 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:03:00.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-04-12T16:03:00.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-04-12T16:03:00.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-04-12T16:03:00.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:03:00.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:03:01.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-04-12T16:03:01.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-04-12T16:03:01.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672972' 2024-04-12T16:03:01.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:03:01.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:03:01.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-04-12T16:03:01.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-04-12T16:03:01.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672972 2-64424509446' 2024-04-12T16:03:01.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:03:01.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:03:01.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-04-12T16:03:01.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:03:01.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-04-12T16:03:01.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:03:01.215 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836497 2024-04-12T16:03:01.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-04-12T16:03:01.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-04-12T16:03:01.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:03:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836497 2024-04-12T16:03:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T16:03:02.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T16:03:02.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:03:03.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-04-12T16:03:03.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:03:03.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-04-12T16:03:03.141 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:03:03.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:03:03.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:03:03.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-04-12T16:03:03.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-04-12T16:03:03.145 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949672972 2024-04-12T16:03:03.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-04-12T16:03:03.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:03:03.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-04-12T16:03:03.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:03:03.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:03:03.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-04-12T16:03:03.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:03:03.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-04-12T16:03:03.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:03:03.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-04-12T16:03:03.572 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509446 2024-04-12T16:03:03.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-04-12T16:03:03.572 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:03:03.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-04-12T16:03:03.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:03:03.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:03:03.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:03:04.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-12T16:03:04.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:03:04.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:03:04.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:03:04.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:03:04.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:03:04.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:03:04.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:03:04.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T16:03:04.923 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:03:04.923 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:03:04.924 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:03:05.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-12T16:03:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:03:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:03:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-04-12T16:03:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-04-12T16:03:06.022 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-04-12T16:03:06.022 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:06.022 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:06.022 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:06.022 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:06.022 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:06.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:06.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-04-12T16:03:06.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:03:06.117 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:06.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:03:06.142 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:05.762+0000 7fe02642d640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:06.142 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:06.002+0000 7fe02642d640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:06.142 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:06.002+0000 7fe02642d640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:06.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-04-12T16:03:06.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-04-12T16:03:06.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-04-12T16:03:06.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:06.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:06.698 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:06.698 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:06.698 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:06.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:06.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-04-12T16:03:06.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:03:06.798 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:06.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:03:06.819 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:06.450+0000 7fe02642d640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:06.820 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:06.678+0000 7fe02642d640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:06.820 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:06.678+0000 7fe02642d640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:06.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-04-12T16:03:06.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-04-12T16:03:07.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-04-12T16:03:07.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:07.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:07.374 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:07.374 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:07.374 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:07.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:07.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-04-12T16:03:07.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:03:07.465 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:07.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:03:07.483 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:07.142+0000 7fe02642d640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:07.483 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:07.354+0000 7fe02642d640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:07.483 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:07.354+0000 7fe02642d640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-04-12T16:03:07.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-04-12T16:03:07.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-04-12T16:03:08.036 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-04-12T16:03:08.036 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:08.036 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:08.036 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:08.036 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:08.036 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:08.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:08.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-04-12T16:03:08.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:03:08.124 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:08.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:03:08.143 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:07.798+0000 7fe02642d640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-04-12T16:03:08.143 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:08.018+0000 7fe02642d640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-04-12T16:03:08.143 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:08.018+0000 7fe02642d640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-04-12T16:03:08.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-04-12T16:03:08.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-04-12T16:03:08.655 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-04-12T16:03:08.655 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:08.655 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:08.655 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:08.656 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:08.656 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:08.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:08.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-04-12T16:03:08.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:03:08.749 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:08.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:03:08.768 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:08.422+0000 7fe02642d640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-04-12T16:03:08.768 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:08.638+0000 7fe02642d640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-04-12T16:03:08.768 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:08.638+0000 7fe02642d640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-04-12T16:03:08.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-04-12T16:03:08.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-04-12T16:03:09.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-04-12T16:03:09.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:09.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:09.360 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:09.360 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:09.360 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:09.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:09.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-04-12T16:03:09.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:03:09.447 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:09.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:03:09.466 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:09.050+0000 7fe02642d640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-04-12T16:03:09.467 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:09.342+0000 7fe02642d640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-04-12T16:03:09.467 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:03:09.342+0000 7fe02642d640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-04-12T16:03:09.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:03:09.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:09.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:03:09.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:03:09.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:03:09.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:03:09.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:03:09.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:03:09.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:03:09.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:03:09.579 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:03:09.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:03:09.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:03:09.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:03:09.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:03:09.581 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:03:09.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:03:09.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:03:09.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:03:09.582 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:03:09.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:03:09.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:03:09.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:03:09.597 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:03:09.597 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:09.597 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:09.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:03:09.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:03:09.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:03:09.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-04-12T16:03:09.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:03:09.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:03:09.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:03:09.601 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:03:09.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:03:09.601 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:03:09.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:03:09.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:03:09.602 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:03:09.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:03:09.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:03:09.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:03:09.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:03:09.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:03:09.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:03:09.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:03:09.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:03:09.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:09.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:09.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:03:09.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:03:09.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:03:09.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:03:09.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:03:09.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:09.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:09.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:03:09.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-04-12T16:03:10.060 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:03:10.060 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:03:10.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:03:10.061 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:03:10.061 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:10.061 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:10.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:03:10.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-04-12T16:03:10.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:03:10.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:03:10.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:03:10.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:03:10.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:03:10.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:03:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:03:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:10.256 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:10.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:03:10.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:03:10.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:03:10.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:03:10.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:03:10.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:03:10.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:03:10.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:03:10.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:10.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:03:10.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:03:10.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:03:10.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:03:10.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:03:11.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:03:11.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:03:11.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:03:11.377 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:03:11.377 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:11.377 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:11.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:03:11.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:03:11.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:03:11.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-04-12T16:03:11.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:11.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:11.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:11.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:11.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:11.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:11.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-04-12T16:03:11.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:03:11.513 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:11.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-04-12T16:03:11.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:11.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:03:11.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:03:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:03:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:11.531 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:03:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:03:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:03:11.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c2f47433-1ac2-40f9-b871-2a7e571a6905 2024-04-12T16:03:11.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c2f47433-1ac2-40f9-b871-2a7e571a6905' 2024-04-12T16:03:11.533 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 c2f47433-1ac2-40f9-b871-2a7e571a6905 2024-04-12T16:03:11.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:03:11.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/WxlmhZK0IBAAY9yDDspFxvOytgYe11sepw== 2024-04-12T16:03:11.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/WxlmhZK0IBAAY9yDDspFxvOytgYe11sepw=="}' 2024-04-12T16:03:11.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c2f47433-1ac2-40f9-b871-2a7e571a6905 -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:03:11.842 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:03:11.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:03:11.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQA/WxlmhZK0IBAAY9yDDspFxvOytgYe11sepw== --osd-uuid c2f47433-1ac2-40f9-b871-2a7e571a6905 2024-04-12T16:03:11.890 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:11.890+0000 7f7d2c64e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:11.910 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:11.910+0000 7f7d2c64e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:11.912 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:11.910+0000 7f7d2c64e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:11.912 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:11.910+0000 7f7d2c64e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:03:11.913 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:11.910+0000 7f7d2c64e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:03:11.913 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:11.910+0000 7f7d2c64e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:03:11.950 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:11.950+0000 7f7d2c64e7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:03:17.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:03:17.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:03:17.750 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:03:17.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:03:17.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:03:18.309 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:03:18.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:03:18.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-04-12T16:03:18.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:03:18.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:03:18.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:03:18.339 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:18.338+0000 7f913eb3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:18.351 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:18.350+0000 7f913eb3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:18.352 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:18.350+0000 7f913eb3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:18.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:03:18.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:03:18.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:03:18.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:03:18.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:03:18.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:18.772 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:03:18.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:03:18.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:18.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:18.907 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:18.906+0000 7f913eb3a7c0 -1 Falling back to public interface 2024-04-12T16:03:19.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:20.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:20.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:20.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:03:20.208 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:03:20.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:20.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:20.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:21.438 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:21.438+0000 7f913eb3a7c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:03:21.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:21.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:21.640 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:03:21.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:03:21.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:21.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:22.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:23.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:23.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:23.141 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:03:23.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:03:23.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:23.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:23.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:24.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:24.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:24.568 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:03:24.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:03:24.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:24.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:24.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:25.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:25.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:25.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:03:25.994 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:03:25.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:25.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:26.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:27.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:27.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:27.595 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:03:27.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:03:27.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:27.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:28.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:29.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:29.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:29.052 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:03:29.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:03:29.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:29.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:30.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:30.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:30.566 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:03:30.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:03:30.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:30.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:30.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:31.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:31.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:31.989 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:03:31.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:03:31.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:31.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:32.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:33.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:33.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:33.952 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:03:33.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:03:33.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:33.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:34.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:35.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:35.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:35.952 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:03:35.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:03:35.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:35.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:36.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:37.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:37.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:37.979 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:03:37.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:03:37.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:37.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:38.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:39.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:39.906 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:03:39.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:39.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:03:39.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:39.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:40.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:41.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:41.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:41.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:03:41.719 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:03:41.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:41.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:42.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:43.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:43.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:43.330 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:03:43.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:03:43.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:43.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:43.763 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1466726735,v1:127.0.0.1:6803/1466726735] [v2:127.0.0.1:6804/1466726735,v1:127.0.0.1:6805/1466726735] exists,up c2f47433-1ac2-40f9-b871-2a7e571a6905 2024-04-12T16:03:43.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:03:43.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:03:43.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:03:43.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-04-12T16:03:43.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:03:43.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:03:43.765 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:43.765 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:43.765 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:43.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:03:43.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-04-12T16:03:43.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:03:43.875 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:43.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:03:43.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:03:43.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:03:43.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:43.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:03:43.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:03:43.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:03:43.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:03:43.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:03:43.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:03:43.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:03:44.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:03:44.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:03:44.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:03:44.005 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:03:44.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:03:44.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:03:44.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:03:44.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:03:44.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:03:44.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:03:44.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:03:44.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:03:44.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:03:44.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:03:44.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:03:44.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:44.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:44.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:03:44.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:03:44.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:03:44.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-04-12T16:03:44.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-04-12T16:03:44.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:44.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:03:44.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:44.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:03:44.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:03:44.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:03:44.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:03:44.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:03:44.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:03:44.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:03:44.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:03:44.020 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:03:44.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:03:44.021 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:03:44.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:03:44.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:03:44.022 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:03:44.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:03:44.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:03:44.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:03:44.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:03:44.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:03:44.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:03:44.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:03:44.026 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:03:44.026 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:44.026 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:44.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:03:44.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:03:44.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:03:44.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:03:44.028 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:03:44.028 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:44.028 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:44.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:03:44.030 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:03:44.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:03:44.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:03:44.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:03:44.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-04-12T16:03:44.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:44.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:03:44.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:03:44.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:03:44.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:03:44.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-04-12T16:03:44.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:03:44.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:03:44.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:03:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:03:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:44.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:03:44.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-04-12T16:03:44.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:03:44.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:03:44.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:03:44.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:03:44.693 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:03:44.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:03:44.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:03:44.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:44.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:44.694 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:44.694 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:44.694 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:44.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:44.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:03:44.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:03:44.811 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:03:44.811 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:03:44.811 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:03:44.811 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:03:44.812 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:03:44.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:03:44.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:03:44.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:44.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:03:44.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:03:44.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:03:44.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:03:44.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:03:45.805 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:03:45.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:03:45.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:03:45.806 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:03:45.806 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:45.806 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:45.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:03:45.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:03:45.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:03:45.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-04-12T16:03:45.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:03:45.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:03:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:03:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:45.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:03:45.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-04-12T16:03:45.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:03:45.958 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:03:45.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-04-12T16:03:45.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:03:45.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:03:45.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:03:45.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:03:45.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:03:45.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:03:45.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:03:45.977 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:03:45.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:03:45.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:03:45.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-04-12T16:03:45.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:03:45.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:03:45.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74fb1f02-e59e-475b-b57a-de537e49132a 2024-04-12T16:03:45.980 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 74fb1f02-e59e-475b-b57a-de537e49132a 2024-04-12T16:03:45.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 74fb1f02-e59e-475b-b57a-de537e49132a' 2024-04-12T16:03:45.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:03:45.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiWxlmVrULABAAd1/wMEWWhj0Mqf0Jy8Jkug== 2024-04-12T16:03:45.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiWxlmVrULABAAd1/wMEWWhj0Mqf0Jy8Jkug=="}' 2024-04-12T16:03:46.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74fb1f02-e59e-475b-b57a-de537e49132a -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:03:46.279 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:03:46.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:03:46.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQBiWxlmVrULABAAd1/wMEWWhj0Mqf0Jy8Jkug== --osd-uuid 74fb1f02-e59e-475b-b57a-de537e49132a 2024-04-12T16:03:46.330 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:46.330+0000 7ff0340e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:46.348 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:46.346+0000 7ff0340e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:46.349 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:46.350+0000 7ff0340e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:46.350 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:46.350+0000 7ff0340e27c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:03:46.350 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:46.350+0000 7ff0340e27c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:03:46.350 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:46.350+0000 7ff0340e27c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:03:46.387 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:46.386+0000 7ff0340e27c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:03:52.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:03:52.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:03:52.149 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:03:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:03:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:03:52.692 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:03:52.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:03:52.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:03:52.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:03:52.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:03:52.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-04-12T16:03:52.722 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:52.722+0000 7f532e26f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:52.743 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:52.742+0000 7f532e26f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:52.744 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:52.742+0000 7f532e26f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:03:53.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:03:53.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:03:53.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:03:53.103 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:03:53.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:03:53.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:03:53.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:53.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:03:53.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:53.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:53.335 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:53.334+0000 7f532e26f7c0 -1 Falling back to public interface 2024-04-12T16:03:53.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:54.514 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:03:54.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:03:54.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:54.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:54.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:55.781 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:03:55.782+0000 7f532e26f7c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:03:55.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:55.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:55.951 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:03:55.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:03:55.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:55.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:56.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:57.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:57.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:57.509 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:03:57.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:03:57.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:57.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:57.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:03:58.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:03:58.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:03:58.952 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:03:58.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:03:58.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:03:58.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:03:59.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:00.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:00.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:00.508 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:04:00.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:04:00.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:00.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:00.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:01.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:01.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:01.947 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:04:01.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:04:01.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:01.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:02.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:03.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:03.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:03.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:04:03.351 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:04:03.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:03.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:03.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:04.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:04.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:04.819 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:04:04.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:04:04.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:04.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:05.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:06.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:06.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:06.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:04:06.231 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:04:06.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:06.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:06.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:07.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:07.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:07.641 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:04:07.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:04:07.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:07.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:08.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:09.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:09.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:09.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:04:09.139 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:04:09.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:09.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:10.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:04:11.086 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:04:11.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:11.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:11.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:12.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:12.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:12.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:04:12.860 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:04:12.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:12.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:13.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:14.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:14.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:14.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:04:14.697 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:04:14.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:14.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:15.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:16.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:16.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:16.529 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:04:16.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:04:16.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:16.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:17.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:18.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:18.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:18.288 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:04:18.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:04:18.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:18.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:18.709 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/795367722,v1:127.0.0.1:6803/795367722] [v2:127.0.0.1:6804/795367722,v1:127.0.0.1:6805/795367722] exists,up 74fb1f02-e59e-475b-b57a-de537e49132a 2024-04-12T16:04:18.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:04:18.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:04:18.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:04:18.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-04-12T16:04:18.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:04:18.710 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:04:18.711 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:04:18.711 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:18.712 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:18.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:04:18.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-04-12T16:04:18.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:04:18.809 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:04:18.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:04:18.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:04:18.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:04:18.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:18.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:04:18.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:04:18.827 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:04:18.827 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:04:18.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:04:18.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:04:18.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:04:18.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:04:18.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:04:18.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:04:18.937 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:04:18.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:04:18.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:04:18.938 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:04:18.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:04:18.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:04:18.939 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:04:18.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:04:18.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:04:18.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:04:18.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:04:18.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:04:18.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:18.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:18.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:04:18.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:04:18.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:04:18.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:04:18.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:04:18.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:04:18.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:04:18.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:04:18.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:04:18.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:04:18.953 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:04:18.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:04:18.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:04:18.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:04:18.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:04:18.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:04:18.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:04:18.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:04:18.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:04:18.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:18.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:18.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:04:18.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:04:18.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:04:18.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:04:18.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:04:18.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:18.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:18.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:04:18.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:04:18.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:04:18.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-04-12T16:04:19.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:04:19.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:04:19.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:04:19.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:04:19.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:19.418 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:19.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:04:19.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-04-12T16:04:19.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:04:19.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:04:19.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:04:19.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:04:19.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:04:19.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:04:19.615 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:04:19.615 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:04:19.615 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:04:19.615 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:04:19.615 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:19.615 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:19.615 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:04:19.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:04:19.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:04:19.735 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:04:19.735 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:04:19.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:04:19.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:04:19.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:04:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:04:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:04:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:04:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:04:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:04:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:04:20.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:04:20.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:04:20.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:04:20.725 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:04:20.726 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:20.726 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:20.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:04:20.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:04:20.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:04:20.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-04-12T16:04:20.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:04:20.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:04:20.762 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:04:20.762 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:20.762 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:20.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:04:20.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-04-12T16:04:20.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:04:20.861 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:04:20.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:04:20.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-04-12T16:04:20.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:04:20.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:04:20.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a8edc9b3-67ba-4486-ad21-98f6bfa2579d 2024-04-12T16:04:20.882 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 a8edc9b3-67ba-4486-ad21-98f6bfa2579d 2024-04-12T16:04:20.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a8edc9b3-67ba-4486-ad21-98f6bfa2579d' 2024-04-12T16:04:20.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:04:20.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEWxlmk26CNRAAS1QtfhH6BUcEZf2GUVn5Yg== 2024-04-12T16:04:20.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEWxlmk26CNRAAS1QtfhH6BUcEZf2GUVn5Yg=="}' 2024-04-12T16:04:20.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a8edc9b3-67ba-4486-ad21-98f6bfa2579d -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:04:21.166 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:04:21.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:04:21.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQCEWxlmk26CNRAAS1QtfhH6BUcEZf2GUVn5Yg== --osd-uuid a8edc9b3-67ba-4486-ad21-98f6bfa2579d 2024-04-12T16:04:21.207 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:21.206+0000 7f78a86a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:21.235 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:21.234+0000 7f78a86a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:21.236 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:21.234+0000 7f78a86a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:21.236 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:21.234+0000 7f78a86a57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:04:21.236 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:21.234+0000 7f78a86a57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:04:21.237 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:21.234+0000 7f78a86a57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:04:21.273 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:21.274+0000 7f78a86a57c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:04:27.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:04:27.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:04:27.145 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:04:27.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:04:27.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:04:27.724 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:04:27.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:04:27.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-04-12T16:04:27.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:04:27.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:04:27.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:04:27.755 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:27.754+0000 7fc685c517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:27.757 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:27.758+0000 7fc685c517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:27.758 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:27.758+0000 7fc685c517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:28.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:04:28.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:04:28.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:04:28.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:04:28.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:04:28.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:28.201 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:04:28.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:04:28.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:28.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:28.327 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:28.326+0000 7fc685c517c0 -1 Falling back to public interface 2024-04-12T16:04:28.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:29.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:29.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:29.651 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:04:29.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:04:29.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:29.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:30.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:30.848 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:30.846+0000 7fc685c517c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:04:31.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:31.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:31.108 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:04:31.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:04:31.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:31.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:31.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:32.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:32.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:32.541 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:04:32.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:04:32.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:32.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:32.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:33.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:33.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:33.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:04:33.943 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:04:33.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:33.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:34.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:35.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:35.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:35.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:04:35.371 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:04:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:35.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:36.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:36.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:36.884 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:04:36.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:04:36.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:36.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:37.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:38.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:38.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:38.347 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:04:38.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:04:38.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:38.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:38.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:39.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:39.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:39.876 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:04:39.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:04:39.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:39.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:40.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:41.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:41.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:41.292 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:04:41.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:04:41.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:41.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:41.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:42.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:42.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:42.819 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:04:42.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:04:42.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:42.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:43.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:44.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:44.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:44.512 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:04:44.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:04:44.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:44.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:45.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:45.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:45.932 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:04:45.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:04:45.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:45.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:46.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:47.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:47.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:47.645 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:04:47.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:04:47.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:47.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:48.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:49.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:49.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:49.321 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:04:49.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:04:49.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:49.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:49.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:50.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:50.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:50.812 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:04:50.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:04:50.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:50.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:51.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:04:52.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:04:52.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:04:52.566 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:04:52.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:04:52.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:04:52.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:04:53.127 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1677292399,v1:127.0.0.1:6803/1677292399] [v2:127.0.0.1:6804/1677292399,v1:127.0.0.1:6805/1677292399] exists,up a8edc9b3-67ba-4486-ad21-98f6bfa2579d 2024-04-12T16:04:53.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:04:53.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:04:53.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:04:53.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-04-12T16:04:53.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:04:53.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:04:53.129 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:04:53.129 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:53.129 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:53.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:04:53.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-04-12T16:04:53.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:04:53.233 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:04:53.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:04:53.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:04:53.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:04:53.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:53.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:04:53.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:04:53.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:04:53.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:04:53.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:04:53.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:04:53.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:04:53.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:04:53.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:04:53.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:04:53.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:04:53.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:04:53.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:04:53.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:04:53.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:04:53.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:04:53.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:04:53.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:04:53.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:04:53.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:04:53.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:04:53.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:04:53.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:53.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:53.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:04:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:04:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:04:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-04-12T16:04:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-04-12T16:04:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:04:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:04:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:04:53.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:04:53.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:04:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:04:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:04:53.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:04:53.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:04:53.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:04:53.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:04:53.384 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:04:53.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:04:53.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:04:53.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:04:53.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:04:53.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:04:53.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:04:53.386 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:04:53.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:04:53.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:04:53.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:04:53.389 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:04:53.389 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:53.389 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:53.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:04:53.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:04:53.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:04:53.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:04:53.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:04:53.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:53.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:53.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:04:53.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-04-12T16:04:53.783 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:04:53.784 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:04:53.784 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:04:53.784 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:04:53.784 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:53.784 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:53.784 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:04:53.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-04-12T16:04:54.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:04:54.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:04:54.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:04:54.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:04:54.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:04:54.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:04:54.011 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:04:54.011 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:04:54.011 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:04:54.012 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:04:54.012 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:54.012 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:54.012 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:04:54.012 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:04:54.012 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:04:54.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:04:54.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:04:54.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:04:54.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:04:54.127 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:04:54.127 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:04:54.127 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:04:54.127 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:04:54.128 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:04:54.128 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:54.128 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:54.128 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:04:54.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:04:54.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:04:54.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:04:54.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:54.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:04:54.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:04:54.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:04:54.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:04:54.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:04:55.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:04:55.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:04:55.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:04:55.168 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:04:55.168 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:55.168 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:55.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:04:55.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:04:55.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:04:55.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-04-12T16:04:55.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:04:55.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:04:55.203 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:04:55.203 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:55.203 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:55.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:04:55.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-04-12T16:04:55.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:04:55.311 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:04:55.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:04:55.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:04:55.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:04:55.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:04:55.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:04:55.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:04:55.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:04:55.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:04:55.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:04:55.328 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:04:55.328 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:04:55.328 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:04:55.328 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-04-12T16:04:55.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:04:55.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:04:55.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf4244dc-4f60-41e1-a81e-6c4cf26ab04c 2024-04-12T16:04:55.330 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 bf4244dc-4f60-41e1-a81e-6c4cf26ab04c 2024-04-12T16:04:55.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bf4244dc-4f60-41e1-a81e-6c4cf26ab04c' 2024-04-12T16:04:55.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:04:55.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnWxlmVJK0FBAAc8Wx6FljBqOxwlLr1gOykQ== 2024-04-12T16:04:55.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnWxlmVJK0FBAAc8Wx6FljBqOxwlLr1gOykQ=="}' 2024-04-12T16:04:55.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf4244dc-4f60-41e1-a81e-6c4cf26ab04c -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:04:55.637 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:04:55.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:04:55.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQCnWxlmVJK0FBAAc8Wx6FljBqOxwlLr1gOykQ== --osd-uuid bf4244dc-4f60-41e1-a81e-6c4cf26ab04c 2024-04-12T16:04:55.679 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:55.678+0000 7f68dd2cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:55.697 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:55.694+0000 7f68dd2cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:55.698 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:55.698+0000 7f68dd2cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:04:55.698 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:55.698+0000 7f68dd2cb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:04:55.699 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:55.698+0000 7f68dd2cb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:04:55.699 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:55.698+0000 7f68dd2cb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:04:55.729 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:04:55.726+0000 7f68dd2cb7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:05:01.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:05:01.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:05:01.624 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:05:01.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:05:01.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:05:02.176 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:05:02.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:05:02.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:05:02.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:05:02.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-04-12T16:05:02.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:05:02.206 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:02.206+0000 7ff3004307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:02.209 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:02.206+0000 7ff3004307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:02.210 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:02.210+0000 7ff3004307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:02.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:05:02.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:05:02.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:05:02.594 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:05:02.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:05:02.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:05:02.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:02.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:05:02.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:02.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:02.779 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:02.777+0000 7ff3004307c0 -1 Falling back to public interface 2024-04-12T16:05:03.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:04.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:04.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:04.061 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:05:04.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:05:04.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:04.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:04.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:05.327 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:05.325+0000 7ff3004307c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:05:05.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:05.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:05.471 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:05:05.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:05:05.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:05.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:05.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:06.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:06.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:06.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:05:06.927 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:05:06.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:06.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:07.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:08.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:08.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:08.334 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:05:08.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:05:08.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:08.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:08.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:09.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:09.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:09.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:05:09.736 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:05:09.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:09.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:10.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:11.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:11.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:11.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:05:11.191 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:05:11.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:11.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:11.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:12.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:12.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:12.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:05:12.607 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:05:12.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:12.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:13.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:05:14.156 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:05:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:14.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:14.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:15.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:15.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:15.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:05:15.589 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:05:15.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:15.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:16.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:17.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:17.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:17.044 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:05:17.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:05:17.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:17.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:17.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:18.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:18.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:18.878 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:05:18.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:05:18.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:18.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:19.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:20.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:20.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:20.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:05:20.892 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:05:20.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:20.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:21.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:22.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:22.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:22.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:05:22.514 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:05:22.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:22.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:23.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:24.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:24.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:24.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:05:24.050 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:05:24.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:24.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:25.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:26.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:26.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:26.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:05:26.234 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:05:26.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:26.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:26.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:27.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:27.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:27.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:05:27.858 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:05:27.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:27.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:28.328 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2459839235,v1:127.0.0.1:6803/2459839235] [v2:127.0.0.1:6804/2459839235,v1:127.0.0.1:6805/2459839235] exists,up bf4244dc-4f60-41e1-a81e-6c4cf26ab04c 2024-04-12T16:05:28.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:05:28.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:05:28.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:05:28.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-04-12T16:05:28.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:05:28.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:05:28.330 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:05:28.330 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:28.330 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:28.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:05:28.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-04-12T16:05:28.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:05:28.440 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:05:28.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:05:28.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:05:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:05:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:05:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:05:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:05:28.459 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:05:28.460 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:05:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:05:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:05:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:05:28.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:05:28.569 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:05:28.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:05:28.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:05:28.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:05:28.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:05:28.571 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:05:28.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:05:28.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:05:28.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:05:28.573 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:05:28.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:05:28.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:05:28.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:05:28.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:05:28.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:28.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:28.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:05:28.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:05:28.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:05:28.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:05:28.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:05:28.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:05:28.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:05:28.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:05:28.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:05:28.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:05:28.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:05:28.586 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:05:28.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:05:28.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:05:28.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:05:28.587 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:05:28.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:05:28.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:05:28.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:05:28.590 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:05:28.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:28.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:28.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:05:28.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:05:28.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:05:28.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:05:28.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:05:28.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:28.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:28.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:05:28.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:05:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:05:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:05:28.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:05:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-04-12T16:05:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:05:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:05:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:05:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:05:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:05:28.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-04-12T16:05:29.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:05:29.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:05:29.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:05:29.053 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:05:29.053 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:29.053 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:29.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:05:29.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:05:29.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:05:29.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:05:29.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:05:29.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:05:29.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:05:29.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:05:29.250 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:05:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:05:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:05:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:05:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:05:29.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:05:29.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:05:29.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:05:29.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:05:29.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:05:29.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:05:29.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:05:29.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:05:29.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-04-12T16:05:29.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:05:29.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:05:29.479 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:05:29.479 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:29.479 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:29.480 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:05:29.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-04-12T16:05:29.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:05:29.577 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:05:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:05:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:05:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:05:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:05:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:05:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:05:29.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:05:30.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:05:30.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:05:30.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:05:30.106 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:05:30.106 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:30.106 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:30.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:05:30.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:05:30.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:05:30.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:05:30.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:05:30.145 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 55b8d197-0b7f-4c79-b771-0ff3a931d868 2024-04-12T16:05:30.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=55b8d197-0b7f-4c79-b771-0ff3a931d868 2024-04-12T16:05:30.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 55b8d197-0b7f-4c79-b771-0ff3a931d868' 2024-04-12T16:05:30.147 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:05:30.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKWxlmCiCdCRAAnFSZpF9J/QeK3YQOS4wW9A== 2024-04-12T16:05:30.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKWxlmCiCdCRAAnFSZpF9J/QeK3YQOS4wW9A=="}' 2024-04-12T16:05:30.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 55b8d197-0b7f-4c79-b771-0ff3a931d868 -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:05:30.467 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:05:30.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:05:30.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKWxlmCiCdCRAAnFSZpF9J/QeK3YQOS4wW9A== --osd-uuid 55b8d197-0b7f-4c79-b771-0ff3a931d868 2024-04-12T16:05:30.516 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:30.513+0000 7fa82fcbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:30.535 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:30.533+0000 7fa82fcbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:30.536 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:30.533+0000 7fa82fcbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:30.537 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:30.533+0000 7fa82fcbb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:05:30.537 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:30.533+0000 7fa82fcbb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:05:30.537 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:30.533+0000 7fa82fcbb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:05:30.566 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:30.565+0000 7fa82fcbb7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:05:36.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:05:36.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:05:36.544 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:05:36.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:05:36.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:05:37.110 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:05:37.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:05:37.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:05:37.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:05:37.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:05:37.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:05:37.133 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:37.129+0000 7ff1c5b997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:37.136 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:37.133+0000 7ff1c5b997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:37.137 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:37.133+0000 7ff1c5b997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:05:37.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:05:37.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:05:37.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:05:37.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:05:37.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:05:37.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:37.529 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:05:37.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:05:37.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:37.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:37.700 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:37.697+0000 7ff1c5b997c0 -1 Falling back to public interface 2024-04-12T16:05:37.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:38.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:38.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:38.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:05:38.991 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:05:38.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:38.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:39.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:40.186 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:05:40.185+0000 7ff1c5b997c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:05:40.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:40.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:40.390 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:05:40.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:05:40.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:40.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:40.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:41.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:41.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:41.906 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:05:41.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:05:41.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:41.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:42.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:43.320 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:05:43.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:05:43.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:43.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:43.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:44.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:44.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:44.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:05:44.740 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:05:44.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:44.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:45.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:46.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:46.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:46.243 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:05:46.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:05:46.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:46.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:46.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:47.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:47.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:47.666 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:05:47.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:05:47.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:47.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:48.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:49.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:49.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:49.084 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:05:49.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:05:49.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:49.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:49.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:50.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:50.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:50.482 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:05:50.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:05:50.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:50.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:50.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:51.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:51.939 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:05:51.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:05:51.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:51.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:52.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:53.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:53.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:53.709 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:05:53.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:05:53.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:53.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:54.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:55.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:55.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:55.315 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:05:55.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:05:55.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:55.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:56.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:57.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:57.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:57.083 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:05:57.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:05:57.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:57.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:57.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:05:58.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:05:58.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:05:58.737 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:05:58.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:05:58.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:05:58.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:05:59.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:00.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:00.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:00.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:06:00.249 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:06:00.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:00.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:01.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:02.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:02.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:02.368 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:06:02.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:06:02.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:02.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:02.806 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2842686931,v1:127.0.0.1:6803/2842686931] [v2:127.0.0.1:6804/2842686931,v1:127.0.0.1:6805/2842686931] exists,up 55b8d197-0b7f-4c79-b771-0ff3a931d868 2024-04-12T16:06:02.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:06:02.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:06:02.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:06:02.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-04-12T16:06:02.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:06:02.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:06:02.808 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:06:02.808 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:02.808 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:02.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:06:02.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-04-12T16:06:02.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:06:02.915 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:06:02.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:06:02.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:06:02.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-04-12T16:06:02.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:02.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:06:02.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:06:02.935 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:06:02.935 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:06:02.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:06:02.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:06:02.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:06:03.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:06:03.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:06:03.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:06:03.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:06:03.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:06:03.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:06:03.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:06:03.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:06:03.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:06:03.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:06:03.252 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:06:03.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:06:03.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:06:03.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:06:03.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:06:03.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:03.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:03.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:06:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:06:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:06:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-04-12T16:06:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-04-12T16:06:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-04-12T16:06:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:06:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:06:03.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:06:03.267 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:06:03.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:06:03.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:06:03.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:06:03.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:06:03.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:06:03.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:06:03.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:06:03.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:06:03.271 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:06:03.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:06:03.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:06:03.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:06:03.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:06:03.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:03.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:03.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:06:03.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:06:03.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:06:03.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:06:03.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:06:03.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:03.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:03.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:06:03.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:06:03.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-04-12T16:06:03.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:06:03.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:06:03.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:06:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:06:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:03.751 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:03.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:06:03.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-04-12T16:06:03.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:06:03.963 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:06:03.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:06:03.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:06:03.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:06:03.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:06:03.964 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:06:03.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:06:03.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:06:03.965 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:06:03.966 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:03.966 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:03.966 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:06:03.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:06:03.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:06:04.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:06:04.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:06:04.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:06:04.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:06:04.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:06:04.084 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:06:04.084 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:06:04.084 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:06:04.085 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:06:04.085 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:04.085 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:04.085 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:06:04.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:06:04.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:06:04.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:06:04.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:04.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:06:04.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:06:04.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:06:04.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:06:04.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:06:05.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:06:05.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:06:05.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:06:05.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:06:05.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:05.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:05.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:06:05.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:06:05.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:06:05.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-04-12T16:06:05.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:06:05.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:06:05.114 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:06:05.114 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:05.114 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:05.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:06:05.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:06:05.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:06:05.213 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:06:05.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-04-12T16:06:05.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:05.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:06:05.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:06:05.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:06:05.234 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-04-12T16:06:05.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:06:05.236 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:06:05.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a8e6aa3-f935-4774-a24f-8dfa541770d3 2024-04-12T16:06:05.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0a8e6aa3-f935-4774-a24f-8dfa541770d3' 2024-04-12T16:06:05.236 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 0a8e6aa3-f935-4774-a24f-8dfa541770d3 2024-04-12T16:06:05.236 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:06:05.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtWxlmISLcDhAAj0mH/nGGJCI0nj0KZWl0Ag== 2024-04-12T16:06:05.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtWxlmISLcDhAAj0mH/nGGJCI0nj0KZWl0Ag=="}' 2024-04-12T16:06:05.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a8e6aa3-f935-4774-a24f-8dfa541770d3 -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:06:05.540 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:06:05.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:06:05.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQDtWxlmISLcDhAAj0mH/nGGJCI0nj0KZWl0Ag== --osd-uuid 0a8e6aa3-f935-4774-a24f-8dfa541770d3 2024-04-12T16:06:05.588 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:05.585+0000 7fb0cbdda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:05.608 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:05.605+0000 7fb0cbdda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:05.609 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:05.605+0000 7fb0cbdda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:05.610 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:05.609+0000 7fb0cbdda7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:06:05.610 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:05.609+0000 7fb0cbdda7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:06:05.610 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:05.609+0000 7fb0cbdda7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:06:05.648 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:05.645+0000 7fb0cbdda7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:06:11.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:06:11.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:06:11.538 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:06:11.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:06:11.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:06:12.093 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:06:12.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:06:12.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:06:12.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-04-12T16:06:12.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:06:12.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:06:12.123 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:12.121+0000 7fa8e0ab67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:12.138 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:12.137+0000 7fa8e0ab67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:12.139 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:12.137+0000 7fa8e0ab67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:12.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:06:12.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:06:12.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:06:12.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:06:12.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:06:12.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:12.543 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:06:12.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:06:12.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:12.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:12.723 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:12.721+0000 7fa8e0ab67c0 -1 Falling back to public interface 2024-04-12T16:06:13.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:14.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:14.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:14.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:06:14.021 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:06:14.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:14.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:14.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:15.263 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:15.261+0000 7fa8e0ab67c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:06:15.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:15.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:15.432 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:06:15.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:06:15.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:15.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:15.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:16.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:16.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:16.970 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:06:16.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:06:16.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:16.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:17.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:18.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:18.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:18.397 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:06:18.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:06:18.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:18.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:18.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:19.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:19.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:19.821 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:06:19.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:06:19.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:19.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:20.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:21.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:21.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:21.365 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:06:21.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:06:21.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:21.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:21.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:22.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:22.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:22.799 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:06:22.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:06:22.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:22.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:23.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:24.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:24.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:24.268 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:06:24.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:06:24.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:24.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:24.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:25.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:25.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:25.677 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:06:25.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:06:25.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:25.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:26.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:27.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:27.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:27.551 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:06:27.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:06:27.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:27.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:28.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:29.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:29.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:29.074 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:06:29.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:06:29.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:29.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:30.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:31.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:31.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:31.105 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:06:31.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:06:31.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:31.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:31.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:32.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:32.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:32.919 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:06:32.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:06:32.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:32.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:33.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:34.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:34.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:34.631 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:06:34.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:06:34.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:34.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:35.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:36.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:36.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:36.766 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:06:36.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:06:36.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:36.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:37.241 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4024651112,v1:127.0.0.1:6803/4024651112] [v2:127.0.0.1:6804/4024651112,v1:127.0.0.1:6805/4024651112] exists,up 0a8e6aa3-f935-4774-a24f-8dfa541770d3 2024-04-12T16:06:37.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:06:37.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:06:37.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:06:37.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-04-12T16:06:37.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:06:37.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:06:37.243 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:06:37.243 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:37.244 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:37.244 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:06:37.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:06:37.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:06:37.341 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:06:37.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:06:37.356 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:06:03.785+0000 7f29e480cc80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-04-12T16:06:37.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:06:37.358 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:06:12.721+0000 7fa8e0ab67c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-04-12T16:06:37.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:06:37.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:37.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:06:37.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:06:37.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:06:37.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:06:37.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:06:37.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:06:37.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:06:37.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:06:37.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:06:37.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:06:37.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:06:37.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:06:37.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:06:37.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:06:37.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:06:37.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:06:37.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:06:37.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:06:37.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:06:37.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:06:37.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:06:37.481 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:06:37.481 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:37.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:37.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:06:37.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:06:37.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:06:37.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:06:37.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:06:37.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:06:37.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:06:37.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:06:37.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:06:37.488 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:06:37.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:06:37.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:06:37.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:06:37.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:06:37.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:06:37.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:06:37.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:06:37.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:06:37.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:37.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:37.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:06:37.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:06:37.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:06:37.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:06:37.494 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:06:37.494 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:37.495 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:37.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:06:37.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:06:37.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:06:37.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:06:37.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:06:37.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-04-12T16:06:37.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:37.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:06:37.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:06:37.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:06:37.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:06:37.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-04-12T16:06:37.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:06:37.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:06:37.960 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:06:37.961 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:06:37.961 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:37.961 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:37.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:06:37.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-04-12T16:06:38.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:06:38.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:06:38.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:06:38.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:06:38.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:06:38.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:06:38.166 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:06:38.166 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:06:38.166 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:06:38.167 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:06:38.167 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:38.167 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:38.167 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:06:38.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:06:38.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:06:38.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:06:38.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:06:38.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:06:38.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:06:38.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:06:38.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:06:38.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:06:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:06:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:06:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:06:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:06:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:06:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:06:39.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:06:39.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:06:39.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:06:39.294 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:06:39.294 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:39.294 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:39.294 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:06:39.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:06:39.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:06:39.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-04-12T16:06:39.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:06:39.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:06:39.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:06:39.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:39.364 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:39.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:06:39.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:06:39.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:06:39.465 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:06:39.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-04-12T16:06:39.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:06:39.481 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:06:39.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:06:39.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:06:39.483 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:06:39.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:06:39.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:06:39.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-04-12T16:06:39.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:06:39.485 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:06:39.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f199b4ab-4c3b-4fae-ba27-9b8d725cca29 2024-04-12T16:06:39.485 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 f199b4ab-4c3b-4fae-ba27-9b8d725cca29 2024-04-12T16:06:39.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f199b4ab-4c3b-4fae-ba27-9b8d725cca29' 2024-04-12T16:06:39.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:06:39.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPXBlmINssHhAAMpGbngLfS7GW717nzglbFQ== 2024-04-12T16:06:39.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPXBlmINssHhAAMpGbngLfS7GW717nzglbFQ=="}' 2024-04-12T16:06:39.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f199b4ab-4c3b-4fae-ba27-9b8d725cca29 -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:06:39.783 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:06:39.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:06:39.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQAPXBlmINssHhAAMpGbngLfS7GW717nzglbFQ== --osd-uuid f199b4ab-4c3b-4fae-ba27-9b8d725cca29 2024-04-12T16:06:39.831 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:39.829+0000 7f40789a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:39.843 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:39.841+0000 7f40789a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:39.844 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:39.841+0000 7f40789a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:39.845 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:39.841+0000 7f40789a67c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:06:39.845 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:39.841+0000 7f40789a67c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:06:39.845 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:39.841+0000 7f40789a67c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:06:39.884 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:39.881+0000 7f40789a67c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:06:45.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:06:45.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:06:45.690 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:06:45.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:06:45.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:06:46.267 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:06:46.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:06:46.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:06:46.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-04-12T16:06:46.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:06:46.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:06:46.297 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:46.293+0000 7f1757ff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:46.324 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:46.321+0000 7f1757ff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:46.325 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:46.321+0000 7f1757ff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:06:46.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:06:46.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:06:46.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:06:46.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:06:46.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:06:46.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:46.720 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:06:46.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:06:46.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:46.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:46.892 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:46.889+0000 7f1757ff77c0 -1 Falling back to public interface 2024-04-12T16:06:47.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:48.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:48.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:48.164 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:06:48.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:06:48.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:48.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:48.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:49.411 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:06:49.409+0000 7f1757ff77c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:06:49.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:49.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:49.570 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:06:49.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:06:49.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:49.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:50.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:51.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:51.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:51.125 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:06:51.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:06:51.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:51.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:51.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:52.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:52.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:52.562 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:06:52.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:06:52.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:52.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:52.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:53.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:53.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:53.983 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:06:53.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:06:53.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:53.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:54.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:55.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:55.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:55.401 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:06:55.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:06:55.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:55.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:56.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:56.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:56.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:06:56.827 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:06:56.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:56.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:57.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:58.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:58.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:58.397 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:06:58.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:06:58.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:58.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:06:58.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:06:59.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:06:59.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:06:59.809 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:06:59.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:06:59.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:06:59.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:00.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:01.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:01.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:01.268 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:07:01.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:07:01.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:01.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:02.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:03.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:03.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:03.087 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:07:03.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:07:03.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:03.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:03.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:04.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:04.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:04.502 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:07:04.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:07:04.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:04.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:05.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:06.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:06.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:06.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:07:06.171 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:07:06.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:06.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:06.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:07.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:07.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:07.920 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:07:07.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:07:07.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:07.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:08.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:09.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:09.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:09.953 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:07:09.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:07:09.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:09.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:10.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:11.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:11.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:11.936 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:07:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:07:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:11.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:12.415 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3088299582,v1:127.0.0.1:6803/3088299582] [v2:127.0.0.1:6804/3088299582,v1:127.0.0.1:6805/3088299582] exists,up f199b4ab-4c3b-4fae-ba27-9b8d725cca29 2024-04-12T16:07:12.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:07:12.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:07:12.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:07:12.416 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-04-12T16:07:12.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:07:12.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:07:12.417 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:07:12.417 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:12.417 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:12.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:07:12.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:07:12.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:07:12.534 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:07:12.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:07:12.551 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:06:37.993+0000 7f2f3bde6c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-04-12T16:07:12.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:07:12.553 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:06:46.889+0000 7f1757ff77c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-04-12T16:07:12.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:07:12.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:12.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:07:12.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:07:12.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:07:12.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:07:12.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:07:12.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:07:12.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:07:12.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:07:12.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:07:12.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:07:12.665 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:07:12.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:07:12.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:07:12.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:07:12.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:07:12.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:07:12.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:07:12.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:07:12.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:07:12.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:07:12.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:07:12.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:07:12.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:12.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:12.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:07:12.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:07:12.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:07:12.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-04-12T16:07:12.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-04-12T16:07:12.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:12.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:07:12.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:12.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:07:12.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:07:12.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:07:12.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:07:12.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:07:12.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:07:12.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:07:12.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:07:12.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:07:12.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:07:12.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:07:12.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:07:12.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:07:12.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:07:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:07:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:07:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:07:12.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:07:12.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:07:12.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:07:12.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:07:12.685 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:07:12.685 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:12.685 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:12.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:07:12.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:07:12.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:07:12.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:07:12.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:07:12.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:12.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:12.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:07:12.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:07:12.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:07:12.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:07:12.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:07:12.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-04-12T16:07:12.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:12.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:07:12.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:07:12.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:07:12.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:07:12.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-04-12T16:07:13.122 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:07:13.122 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:07:13.122 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:07:13.123 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:07:13.123 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:13.123 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:13.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:07:13.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-04-12T16:07:13.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:07:13.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:07:13.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:07:13.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:07:13.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:07:13.335 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:07:13.335 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:07:13.335 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:07:13.336 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:07:13.336 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:07:13.336 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:13.336 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:13.336 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:07:13.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:07:13.337 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:07:13.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:07:13.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:07:13.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:07:13.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:07:13.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:07:13.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:07:13.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:07:13.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:07:13.449 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:07:13.449 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:13.450 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:13.451 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:07:13.451 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:07:13.451 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:07:13.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:07:13.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:13.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:07:13.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:07:13.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:07:13.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:07:13.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:07:14.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:07:14.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:07:14.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:07:14.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:07:14.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:14.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:14.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:07:14.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:07:14.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:07:14.483 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-04-12T16:07:14.483 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:07:14.483 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:07:14.483 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:07:14.483 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:14.484 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:14.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:07:14.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:07:14.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:07:14.591 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:07:14.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-04-12T16:07:14.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:07:14.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:07:14.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:07:14.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:07:14.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:07:14.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-04-12T16:07:14.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:07:14.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:07:14.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d41280a-c47e-4964-8195-b097ce458c10 2024-04-12T16:07:14.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8d41280a-c47e-4964-8195-b097ce458c10' 2024-04-12T16:07:14.612 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 8d41280a-c47e-4964-8195-b097ce458c10 2024-04-12T16:07:14.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:07:14.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyXBlmTrNiJRAAM0mwaJ+k26V9PkHQTVwSCg== 2024-04-12T16:07:14.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyXBlmTrNiJRAAM0mwaJ+k26V9PkHQTVwSCg=="}' 2024-04-12T16:07:14.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d41280a-c47e-4964-8195-b097ce458c10 -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:07:14.911 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:07:14.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:07:14.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQAyXBlmTrNiJRAAM0mwaJ+k26V9PkHQTVwSCg== --osd-uuid 8d41280a-c47e-4964-8195-b097ce458c10 2024-04-12T16:07:14.966 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:14.960+0000 7f745c8fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:14.980 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:14.976+0000 7f745c8fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:14.981 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:14.976+0000 7f745c8fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:14.981 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:14.976+0000 7f745c8fb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:07:14.982 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:14.976+0000 7f745c8fb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:07:14.982 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:14.976+0000 7f745c8fb7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:07:15.020 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:15.016+0000 7f745c8fb7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:07:20.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:07:20.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:07:20.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:07:20.888 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:07:20.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:07:21.465 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:07:21.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:07:21.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-04-12T16:07:21.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:07:21.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:07:21.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:07:21.495 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:21.492+0000 7f96d2a407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:21.498 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:21.492+0000 7f96d2a407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:21.499 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:21.496+0000 7f96d2a407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:21.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:07:21.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:07:21.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:07:21.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:07:21.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:07:21.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:21.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:07:21.885 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:07:21.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:21.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:22.041 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:22.036+0000 7f96d2a407c0 -1 Falling back to public interface 2024-04-12T16:07:22.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:23.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:23.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:23.305 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:07:23.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:07:23.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:23.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:23.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:24.524 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:24.520+0000 7f96d2a407c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:07:24.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:24.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:24.703 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:07:24.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:07:24.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:24.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:25.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:26.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:26.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:26.230 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:07:26.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:07:26.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:26.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:26.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:27.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:27.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:27.670 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:07:27.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:07:27.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:27.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:28.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:29.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:29.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:29.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:07:29.091 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:07:29.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:29.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:29.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:30.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:30.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:30.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:07:30.609 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:07:30.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:30.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:31.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:32.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:32.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:32.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:07:32.033 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:07:32.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:32.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:32.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:33.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:33.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:33.578 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:07:33.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:07:33.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:33.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:34.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:35.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:35.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:35.040 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:07:35.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:07:35.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:35.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:35.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:36.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:36.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:36.555 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:07:36.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:07:36.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:36.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:37.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:38.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:38.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:38.147 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:07:38.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:07:38.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:38.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:39.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:40.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:40.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:40.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:07:40.120 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:07:40.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:40.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:40.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:41.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:41.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:41.636 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:07:41.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:07:41.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:41.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:42.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:43.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:43.417 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:07:43.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:43.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:07:43.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:43.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:44.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:45.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:45.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:45.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:07:45.388 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:07:45.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:45.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:46.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:47.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:47.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:47.271 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:07:47.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:07:47.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:47.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:47.688 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3226345587,v1:127.0.0.1:6803/3226345587] [v2:127.0.0.1:6804/3226345587,v1:127.0.0.1:6805/3226345587] exists,up 8d41280a-c47e-4964-8195-b097ce458c10 2024-04-12T16:07:47.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:07:47.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:07:47.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:07:47.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-04-12T16:07:47.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:07:47.689 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:07:47.690 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:07:47.690 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:47.690 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:47.690 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:07:47.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:07:47.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:07:47.801 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:07:47.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:07:47.817 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:07:13.156+0000 7f87c1716c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-04-12T16:07:47.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:07:47.820 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:07:22.036+0000 7f96d2a407c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-04-12T16:07:47.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:07:47.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:47.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:07:47.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:07:47.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:07:47.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:07:47.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:07:47.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:07:47.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:07:47.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:07:47.930 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:07:47.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:07:47.931 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:07:47.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:07:47.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:07:47.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:07:47.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:07:47.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:07:47.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:07:47.935 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:07:47.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:07:47.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:07:47.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:07:47.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:07:47.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:47.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:47.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:07:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:07:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:07:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-04-12T16:07:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-04-12T16:07:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:07:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:47.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:07:47.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:07:47.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:07:47.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:07:47.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:07:47.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:07:47.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:07:47.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:07:47.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:07:47.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:07:47.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:07:47.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:07:47.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:07:47.948 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:07:47.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:07:47.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:07:47.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:07:47.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:07:47.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:07:47.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:07:47.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:07:47.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:07:47.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:47.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:47.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:07:47.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:07:47.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:07:47.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:07:47.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:07:47.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:47.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:47.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:07:47.956 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:07:47.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:07:47.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:07:47.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:07:47.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-04-12T16:07:47.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:47.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:07:47.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:07:47.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:07:47.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:07:47.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-04-12T16:07:48.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:07:48.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:07:48.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:07:48.413 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:07:48.413 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:48.413 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:48.414 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:07:48.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-04-12T16:07:48.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:07:48.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:07:48.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:07:48.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:07:48.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:07:48.626 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:07:48.626 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:07:48.626 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:07:48.626 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:07:48.627 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:07:48.627 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:48.628 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:48.628 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:07:48.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:07:48.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:07:48.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:07:48.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:07:48.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:07:48.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:07:48.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:07:48.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:07:48.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:07:48.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:07:48.741 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:07:48.742 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:48.742 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:48.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:07:48.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:07:48.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:07:48.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:07:48.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:48.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:07:48.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:07:48.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:07:48.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:07:48.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:07:49.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:07:49.729 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:07:49.729 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:07:49.729 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:07:49.729 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:49.729 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:49.730 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:07:49.730 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:07:49.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:07:49.765 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-04-12T16:07:49.765 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:07:49.765 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:07:49.765 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:07:49.766 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:49.766 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:49.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:07:49.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:07:49.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:07:49.858 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:07:49.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-04-12T16:07:49.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:07:49.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:07:49.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:07:49.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:07:49.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:07:49.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:07:49.873 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:07:49.874 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:07:49.874 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:07:49.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:07:49.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:07:49.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:07:49.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:07:49.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:07:49.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:07:49.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:07:49.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:07:49.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:07:49.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:07:49.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:07:49.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:07:49.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-04-12T16:07:49.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:07:49.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:07:49.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=940a8e2d-a6ca-44b6-8ed8-7ecf0841896c 2024-04-12T16:07:49.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 940a8e2d-a6ca-44b6-8ed8-7ecf0841896c' 2024-04-12T16:07:49.880 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 940a8e2d-a6ca-44b6-8ed8-7ecf0841896c 2024-04-12T16:07:49.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:07:49.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVXBlmlU8UNRAAKSQJEz4ewiBLh/xrfSEB7g== 2024-04-12T16:07:49.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVXBlmlU8UNRAAKSQJEz4ewiBLh/xrfSEB7g=="}' 2024-04-12T16:07:49.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 940a8e2d-a6ca-44b6-8ed8-7ecf0841896c -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:07:50.177 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:07:50.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:07:50.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQBVXBlmlU8UNRAAKSQJEz4ewiBLh/xrfSEB7g== --osd-uuid 940a8e2d-a6ca-44b6-8ed8-7ecf0841896c 2024-04-12T16:07:50.231 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:50.228+0000 7f694fb867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:50.246 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:50.240+0000 7f694fb867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:50.247 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:50.244+0000 7f694fb867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:50.248 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:50.244+0000 7f694fb867c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:07:50.248 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:50.244+0000 7f694fb867c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:07:50.248 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:50.244+0000 7f694fb867c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:07:50.285 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:50.280+0000 7f694fb867c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:07:55.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:07:55.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:07:55.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:07:55.984 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:07:55.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:07:56.544 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:07:56.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:07:56.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-04-12T16:07:56.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:07:56.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:07:56.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:07:56.575 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:56.572+0000 7f4497f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:56.577 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:56.572+0000 7f4497f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:56.578 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:56.572+0000 7f4497f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:07:57.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:07:57.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:07:57.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:07:57.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:07:57.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:07:57.006 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:07:57.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:57.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:07:57.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:57.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:57.129 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:57.124+0000 7f4497f7d7c0 -1 Falling back to public interface 2024-04-12T16:07:57.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:58.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:58.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:58.422 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:07:58.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:07:58.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:58.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:07:58.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:07:59.612 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:07:59.608+0000 7f4497f7d7c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:07:59.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:07:59.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:07:59.886 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:07:59.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:07:59.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:07:59.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:00.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:01.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:01.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:01.342 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:08:01.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:08:01.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:01.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:01.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:02.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:02.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:02.783 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:08:02.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:08:02.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:02.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:03.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:04.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:04.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:04.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:08:04.193 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:08:04.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:04.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:04.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:05.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:05.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:05.650 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:08:05.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:08:05.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:05.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:06.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:07.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:07.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:07.054 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:08:07.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:08:07.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:07.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:07.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:08.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:08.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:08.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:08:08.501 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:08:08.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:08.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:08.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:09.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:09.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:09.946 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:08:09.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:08:09.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:09.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:10.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:11.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:11.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:11.441 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:08:11.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:08:11.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:11.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:12.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:13.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:13.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:13.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:08:13.302 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:08:13.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:13.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:13.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:14.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:14.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:14.780 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:08:14.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:08:14.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:14.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:15.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:16.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:16.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:16.553 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:08:16.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:08:16.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:16.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:17.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:18.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:08:18.647 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:08:18.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:18.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:20.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:21.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:21.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:21.069 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:08:21.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:08:21.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:21.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:21.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:22.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:22.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:22.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:08:22.496 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:08:22.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:22.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:22.958 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1994213094,v1:127.0.0.1:6803/1994213094] [v2:127.0.0.1:6804/1994213094,v1:127.0.0.1:6805/1994213094] exists,up 940a8e2d-a6ca-44b6-8ed8-7ecf0841896c 2024-04-12T16:08:22.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:08:22.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:08:22.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:08:22.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-04-12T16:08:22.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:08:22.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:08:22.961 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:08:22.962 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:22.962 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:22.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:08:22.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:08:22.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:08:23.072 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:08:23.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:08:23.089 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:07:48.444+0000 7fb5fa306c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-04-12T16:08:23.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:08:23.090 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:07:57.128+0000 7f4497f7d7c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-04-12T16:08:23.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:08:23.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:23.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:08:23.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:08:23.092 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:08:23.092 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:08:23.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:08:23.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:08:23.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:08:23.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:08:23.201 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:08:23.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:08:23.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:08:23.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:08:23.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:08:23.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:08:23.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:08:23.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:08:23.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:08:23.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:08:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:08:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:08:23.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:08:23.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:08:23.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:23.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:23.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:08:23.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:08:23.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:08:23.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-04-12T16:08:23.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-04-12T16:08:23.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:23.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:08:23.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:23.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:08:23.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:08:23.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:08:23.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:08:23.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:08:23.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:08:23.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:08:23.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:08:23.219 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:08:23.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:08:23.219 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:08:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:08:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:08:23.223 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:08:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:08:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:08:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:08:23.223 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:08:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:08:23.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:08:23.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:08:23.224 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:08:23.224 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:23.224 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:23.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:08:23.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:08:23.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:08:23.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:08:23.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:08:23.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:23.226 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:23.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:08:23.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-04-12T16:08:23.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:08:23.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:08:23.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:08:23.700 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:08:23.700 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:23.700 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:23.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:08:23.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-04-12T16:08:23.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:08:23.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:08:23.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:08:23.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:08:23.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:08:23.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:08:23.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:08:24.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:08:24.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:08:24.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:08:24.014 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:08:24.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:08:24.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:08:24.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:08:24.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:24.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:08:24.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:08:24.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:08:24.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:08:24.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:08:25.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:08:25.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:08:25.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:08:25.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:08:25.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:25.016 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:25.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:08:25.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:08:25.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:08:25.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-04-12T16:08:25.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:08:25.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:08:25.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:08:25.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:25.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:25.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:08:25.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:08:25.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:08:25.147 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:08:25.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:08:25.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:08:25.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:08:25.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:08:25.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:08:25.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:08:25.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:08:25.163 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:08:25.164 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:08:25.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:08:25.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:08:25.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-04-12T16:08:25.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:08:25.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:08:25.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4343cb5a-8c2c-4f23-a002-fb26b4243a6e 2024-04-12T16:08:25.167 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 4343cb5a-8c2c-4f23-a002-fb26b4243a6e 2024-04-12T16:08:25.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4343cb5a-8c2c-4f23-a002-fb26b4243a6e' 2024-04-12T16:08:25.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:08:25.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5XBlmfZ7aChAAGCF7hWnWCSdfZPf0vbf3sA== 2024-04-12T16:08:25.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5XBlmfZ7aChAAGCF7hWnWCSdfZPf0vbf3sA=="}' 2024-04-12T16:08:25.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4343cb5a-8c2c-4f23-a002-fb26b4243a6e -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:08:25.455 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:08:25.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:08:25.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQB5XBlmfZ7aChAAGCF7hWnWCSdfZPf0vbf3sA== --osd-uuid 4343cb5a-8c2c-4f23-a002-fb26b4243a6e 2024-04-12T16:08:25.505 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:25.500+0000 7f6241cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:08:25.559 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:25.556+0000 7f6241cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:08:25.561 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:25.556+0000 7f6241cf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:08:25.562 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:25.556+0000 7f6241cf57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:08:25.562 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:25.556+0000 7f6241cf57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:08:25.563 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:25.556+0000 7f6241cf57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:08:25.639 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:25.636+0000 7f6241cf57c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:08:31.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:08:31.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:08:31.320 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:08:31.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:08:31.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:08:31.877 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:08:31.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:08:31.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:08:31.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-04-12T16:08:31.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:08:31.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:08:31.907 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:31.900+0000 7f933386c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:08:31.922 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:31.916+0000 7f933386c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:08:31.924 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:31.920+0000 7f933386c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:08:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:08:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:08:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:08:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:08:32.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:08:32.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:32.313 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:08:32.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:08:32.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:32.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:32.480 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:32.476+0000 7f933386c7c0 -1 Falling back to public interface 2024-04-12T16:08:32.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:33.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:33.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:33.762 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:08:33.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:08:33.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:33.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:34.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:35.028 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:08:35.024+0000 7f933386c7c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:08:35.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:35.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:35.168 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:08:35.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:08:35.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:35.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:35.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:36.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:36.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:36.656 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:08:36.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:08:36.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:36.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:37.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:38.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:38.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:38.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:08:38.072 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:08:38.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:38.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:38.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:39.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:39.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:39.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:08:39.490 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:08:39.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:39.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:39.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:40.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:40.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:40.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:08:40.947 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:08:40.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:40.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:41.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:42.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:42.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:42.378 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:08:42.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:08:42.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:42.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:42.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:43.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:43.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:43.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:08:43.889 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:08:43.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:43.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:44.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:45.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:45.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:45.317 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:08:45.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:08:45.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:45.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:45.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:46.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:46.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:46.818 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:08:46.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:08:46.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:46.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:47.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:48.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:48.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:48.607 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:08:48.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:08:48.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:48.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:49.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:50.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:50.093 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:08:50.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:50.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:08:50.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:50.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:51.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:52.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:52.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:52.044 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:08:52.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:08:52.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:52.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:52.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:53.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:53.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:53.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:08:53.779 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:08:53.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:53.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:54.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:55.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:55.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:55.448 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:08:55.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:08:55.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:55.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:56.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:08:57.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:08:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:08:57.583 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:08:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:08:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:08:57.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:08:58.018 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2809841649,v1:127.0.0.1:6803/2809841649] [v2:127.0.0.1:6804/2809841649,v1:127.0.0.1:6805/2809841649] exists,up 4343cb5a-8c2c-4f23-a002-fb26b4243a6e 2024-04-12T16:08:58.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:08:58.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:08:58.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:08:58.020 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-04-12T16:08:58.020 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:08:58.020 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:08:58.021 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:08:58.021 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:58.021 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:58.021 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:08:58.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:08:58.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:08:58.129 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:08:58.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:08:58.145 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:08:23.736+0000 7f88117a5c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-04-12T16:08:58.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:08:58.146 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:08:32.476+0000 7f933386c7c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-04-12T16:08:58.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:08:58.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:58.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:08:58.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:08:58.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:08:58.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:08:58.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:08:58.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:08:58.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:08:58.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:08:58.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:08:58.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:08:58.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:08:58.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:08:58.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:08:58.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:08:58.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:08:58.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:08:58.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:08:58.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:08:58.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:08:58.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:08:58.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:08:58.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:08:58.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:58.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:58.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:08:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:08:58.272 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:08:58.272 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:08:58.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:08:58.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:08:58.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:08:58.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:08:58.273 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:08:58.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:08:58.274 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:08:58.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:08:58.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:08:58.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:08:58.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:08:58.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:08:58.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:08:58.277 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:08:58.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:08:58.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:08:58.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:08:58.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:08:58.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:58.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:58.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:08:58.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:08:58.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:08:58.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-04-12T16:08:58.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:08:58.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:58.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:58.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20222 2024-04-12T16:08:58.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:08:58.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:08:58.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:08:58.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-04-12T16:08:58.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-04-12T16:08:58.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:58.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:08:58.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:08:58.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:08:58.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-04-12T16:08:58.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-04-12T16:08:58.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:08:58.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:08:58.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:08:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:08:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:58.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:08:58.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-04-12T16:08:58.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:08:58.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:08:58.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:08:58.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:08:58.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:08:58.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:08:58.954 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:08:58.954 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:08:58.954 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:08:58.954 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:08:58.954 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:58.955 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:58.955 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:08:58.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:08:58.955 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get fsid 2024-04-12T16:08:59.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:08:59.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:08:59.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:08:59.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:08:59.068 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:08:59.068 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:08:59.068 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:08:59.068 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:08:59.069 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:08:59.069 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:08:59.069 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:08:59.069 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:08:59.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:08:59.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20222/ceph-mon.a.asok config get mon_host 2024-04-12T16:08:59.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-04-12T16:08:59.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-04-12T16:08:59.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:08:59.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:08:59.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:08:59.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-04-12T16:08:59.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:09:00.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:09:00.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:09:00.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:09:00.088 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:09:00.088 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:00.089 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:09:00.089 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:09:00.090 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:09:00.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:09:00.127 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-04-12T16:09:00.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:09:00.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:09:00.128 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:09:00.128 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:00.128 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:09:00.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-mon.a.asok 2024-04-12T16:09:00.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:09:00.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-mon.a.asok log flush 2024-04-12T16:09:00.226 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:09:00.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:09:00.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-04-12T16:09:00.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-04-12T16:09:00.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:09:00.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:09:00.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-04-12T16:09:00.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:09:00.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:09:00.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:09:00.245 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:09:00.245 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:09:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:09:00.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-04-12T16:09:00.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-04-12T16:09:00.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:09:00.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=19cba438-6926-4e73-95f4-67d4b9ec7c61 2024-04-12T16:09:00.247 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 19cba438-6926-4e73-95f4-67d4b9ec7c61 2024-04-12T16:09:00.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 19cba438-6926-4e73-95f4-67d4b9ec7c61' 2024-04-12T16:09:00.248 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:09:00.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcXBlmV3PNDxAAUut5ueOP8+R75JpX+m2x9g== 2024-04-12T16:09:00.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcXBlmV3PNDxAAUut5ueOP8+R75JpX+m2x9g=="}' 2024-04-12T16:09:00.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 19cba438-6926-4e73-95f4-67d4b9ec7c61 -i td/test-erasure-code-plugins/0/new.json 2024-04-12T16:09:00.544 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:09:00.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-04-12T16:09:00.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQCcXBlmV3PNDxAAUut5ueOP8+R75JpX+m2x9g== --osd-uuid 19cba438-6926-4e73-95f4-67d4b9ec7c61 2024-04-12T16:09:00.595 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:00.587+0000 7fb3436987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:00.613 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:00.607+0000 7fb3436987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:00.614 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:00.607+0000 7fb3436987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:00.615 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:00.607+0000 7fb3436987c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:09:00.615 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:00.607+0000 7fb3436987c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:09:00.615 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:00.607+0000 7fb3436987c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-04-12T16:09:00.653 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:00.647+0000 7fb3436987c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-04-12T16:09:06.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-04-12T16:09:06.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:09:06.459 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:09:06.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:09:06.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:09:07.019 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:09:07.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:09:07.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:09:07.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a5032c6-ebc1-4768-85fe-4140727228ec --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20222/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-04-12T16:09:07.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:09:07.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:09:07.049 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:07.047+0000 7f503e18e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:07.070 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:07.067+0000 7f503e18e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:07.071 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:07.071+0000 7f503e18e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:07.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:09:07.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:09:07.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:09:07.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:09:07.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:09:07.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:07.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:09:07.402 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:09:07.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:07.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:07.613 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:07.611+0000 7f503e18e7c0 -1 Falling back to public interface 2024-04-12T16:09:07.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:08.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:08.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:08.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:09:08.830 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:09:08.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:08.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:09.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:10.102 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:10.099+0000 7f503e18e7c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:09:10.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:10.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:10.234 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:09:10.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:09:10.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:10.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:10.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:11.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:11.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:11.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:09:11.700 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:09:11.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:11.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:12.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:13.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:13.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:13.180 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:09:13.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:09:13.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:13.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:13.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:14.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:14.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:14.624 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:09:14.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:09:14.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:14.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:15.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:16.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:16.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:16.105 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:09:16.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:09:16.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:16.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:16.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:17.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:17.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:17.526 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:09:17.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:09:17.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:17.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:17.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:18.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:18.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:18.966 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:09:18.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:09:18.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:18.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:19.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:20.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:20.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:20.396 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:09:20.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:09:20.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:20.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:20.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:21.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:21.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:21.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:09:21.855 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:09:21.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:21.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:22.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:23.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:23.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:23.784 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:09:23.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:09:23.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:23.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:24.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:25.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:25.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:25.814 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:09:25.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:09:25.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:25.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:26.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:27.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:27.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:27.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:09:27.823 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:09:27.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:27.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:28.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:29.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:29.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:29.356 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:09:29.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:09:29.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:29.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:30.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:31.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:31.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:31.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:09:31.251 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:09:31.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:31.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:31.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:32.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:32.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:32.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:09:32.711 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:09:32.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:32.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:33.166 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/639411100,v1:127.0.0.1:6803/639411100] [v2:127.0.0.1:6804/639411100,v1:127.0.0.1:6805/639411100] exists,up 19cba438-6926-4e73-95f4-67d4b9ec7c61 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:09:33.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20222/ceph-osd.0.asok 2024-04-12T16:09:33.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-04-12T16:09:33.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20222/ceph-osd.0.asok log flush 2024-04-12T16:09:33.277 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:09:33.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-04-12T16:09:33.294 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:08:58.787+0000 7fc6ecda6c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-04-12T16:09:33.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-04-12T16:09:33.295 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:09:07.611+0000 7f503e18e7c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-04-12T16:09:33.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-04-12T16:09:33.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:09:33.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:09:33.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:09:33.297 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:09:33.297 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:09:33.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:09:33.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:09:33.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:09:33.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:09:33.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:09:33.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:09:33.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:09:33.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:09:33.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:09:33.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:09:33.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:09:33.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:09:33.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:09:33.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:09:33.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:09:33.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:09:33.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:09:33.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:09:33.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:33.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:09:33.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-code-plugins 0 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:09:33.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:09:33.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:09:33.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:09:33.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:09:33.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:09:33.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:09:33.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:09:33.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:09:33.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:09:33.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:09:33.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:09:33.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:09:33.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:09:33.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:09:33.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:09:33.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-12T16:09:33.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-04-12T16:09:33.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:09:33.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:33.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20222 2024-04-12T16:09:33.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20222 2024-04-12T16:09:33.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:09:33.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:09:33.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-04-12T16:09:33.638 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-12T16:09:33.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-12T16:09:33.693 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-04-12T16:09:33.693 DEBUG:teuthology.orchestra.run.smithi137:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf9c5618cc018927fa77780d25b6deb1f5dc254d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-04-12T16:09:33.747 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-12T16:09:33.755 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-12T16:09:33.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-12T16:09:33.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-12T16:09:33.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-code 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none ' 2024-04-12T16:09:33.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:09:33.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:09:33.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:09:33.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:09:33.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:09:33.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:09:33.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:09:33.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:09:33.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:09:33.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:09:33.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:09:33.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:09:33.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:09:33.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:09:33.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:09:33.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-04-12T16:09:33.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:09:33.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:33.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:09:33.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48107 2024-04-12T16:09:33.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:09:33.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:09:33.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2024-04-12T16:09:33.769 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:09:33.769 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:33.769 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:09:33.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48107 2024-04-12T16:09:33.770 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:09:33.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-12T16:09:33.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2024-04-12T16:09:33.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-04-12T16:09:34.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:09:34.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:09:34.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:09:34.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:09:34.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:34.251 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:09:34.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:09:34.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:09:34.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:09:34.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:09:34.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:09:34.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:09:34.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:09:34.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:09:34.463 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:09:34.463 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:09:34.463 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:09:34.464 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:09:34.464 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:34.464 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:09:34.464 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48107/ceph-mon.a.asok 2024-04-12T16:09:34.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:09:34.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48107/ceph-mon.a.asok config get fsid 2024-04-12T16:09:34.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48107/ceph-mon.a.asok 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:09:34.584 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48107/ceph-mon.a.asok config get mon_host 2024-04-12T16:09:34.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-04-12T16:09:34.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code 2024-04-12T16:09:34.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:09:34.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:09:34.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:09:34.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code/x 2024-04-12T16:09:34.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:09:35.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:09:35.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:09:35.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:09:35.598 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:09:35.598 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:35.598 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:09:35.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:09:35.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:09:35.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:09:35.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-04-12T16:09:35.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:09:35.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:09:35.633 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:09:35.633 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:35.633 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:09:35.633 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48107/ceph-mon.a.asok 2024-04-12T16:09:35.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-04-12T16:09:35.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.48107/ceph-mon.a.asok log flush 2024-04-12T16:09:35.731 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:09:35.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-04-12T16:09:35.750 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:09:34.291+0000 7f6a21973c80 0 load: jerasure load: lrc 2024-04-12T16:09:35.753 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-04-12T16:09:35.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:09:35.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/0 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:09:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:09:35.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/0 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=218af436-86bc-4625-ba11-38ccb7371b8f 2024-04-12T16:09:35.756 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 218af436-86bc-4625-ba11-38ccb7371b8f 2024-04-12T16:09:35.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 218af436-86bc-4625-ba11-38ccb7371b8f' 2024-04-12T16:09:35.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:09:35.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/XBlm0Fs8LhAA3Yvd0WTWOPYkZ+fA3v9FAg== 2024-04-12T16:09:35.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/XBlm0Fs8LhAA3Yvd0WTWOPYkZ+fA3v9FAg=="}' 2024-04-12T16:09:35.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 218af436-86bc-4625-ba11-38ccb7371b8f -i td/test-erasure-code/0/new.json 2024-04-12T16:09:36.056 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:09:36.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/0/new.json 2024-04-12T16:09:36.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/XBlm0Fs8LhAA3Yvd0WTWOPYkZ+fA3v9FAg== --osd-uuid 218af436-86bc-4625-ba11-38ccb7371b8f 2024-04-12T16:09:36.106 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:36.103+0000 7fc46d1b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:36.116 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:36.115+0000 7fc46d1b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:36.117 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:36.115+0000 7fc46d1b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:36.118 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:36.115+0000 7fc46d1b07c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-04-12T16:09:36.118 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:36.115+0000 7fc46d1b07c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-04-12T16:09:36.118 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:36.115+0000 7fc46d1b07c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-04-12T16:09:36.158 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:36.155+0000 7fc46d1b07c0 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-04-12T16:09:42.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-04-12T16:09:42.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:09:42.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:09:42.147 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:09:42.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:09:42.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:09:42.719 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:09:42.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:09:42.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:09:42.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:09:42.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:09:42.750 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:42.747+0000 7f02e54fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:42.772 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:42.771+0000 7f02e54fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:42.774 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:42.771+0000 7f02e54fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:09:43.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:09:43.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:09:43.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:09:43.128 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:09:43.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:09:43.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:09:43.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:43.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:09:43.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:43.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:43.322 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:43.319+0000 7f02e54fe7c0 -1 Falling back to public interface 2024-04-12T16:09:43.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:44.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:44.588 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:44.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:45.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:45.838 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:09:45.835+0000 7f02e54fe7c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:09:46.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:46.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:46.037 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:09:46.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:09:46.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:46.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:46.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:47.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:47.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:47.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:09:47.567 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:09:47.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:47.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:47.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:48.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:48.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:48.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:09:48.988 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:09:48.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:48.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:49.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:50.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:50.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:50.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:09:50.407 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:09:50.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:50.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:50.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:51.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:51.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:51.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:09:51.877 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:09:51.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:51.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:52.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:53.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:53.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:53.309 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:09:53.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:09:53.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:53.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:53.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:54.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:54.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:54.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:09:54.925 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:09:54.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:54.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:55.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:56.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:56.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:56.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:09:56.360 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:09:56.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:56.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:57.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:58.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:58.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:58.053 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:09:58.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:09:58.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:58.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:09:58.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:09:59.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:09:59.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:09:59.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:09:59.611 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:09:59.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:09:59.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:10:00.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:01.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:01.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:01.051 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:10:01.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:10:01.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:10:01.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:01.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:02.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:02.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:02.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:10:02.916 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:10:02.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:02.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:10:03.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:04.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:04.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:04.843 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:10:04.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:10:04.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:04.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:10:05.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:06.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:06.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:06.997 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:10:06.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:10:06.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:06.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:10:07.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:08.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:08.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:08.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:10:08.586 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:10:08.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:08.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:10:09.064 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/457955571,v1:127.0.0.1:6803/457955571] [v2:127.0.0.1:6804/457955571,v1:127.0.0.1:6805/457955571] exists,up 218af436-86bc-4625-ba11-38ccb7371b8f 2024-04-12T16:10:09.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:10:09.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:10:09.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:10:09.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:10:09.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/1 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:10:09.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:10:09.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:10:09.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:10:09.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:10:09.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:10:09.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:10:09.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:10:09.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/1 2024-04-12T16:10:09.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:10:09.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00ab472f-638f-484a-885d-eab340012a93 2024-04-12T16:10:09.072 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 00ab472f-638f-484a-885d-eab340012a93 2024-04-12T16:10:09.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 00ab472f-638f-484a-885d-eab340012a93' 2024-04-12T16:10:09.072 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:10:09.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhXBlmzwYHBRAAgzGnug3djzJknN/wLueCaQ== 2024-04-12T16:10:09.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhXBlmzwYHBRAAgzGnug3djzJknN/wLueCaQ=="}' 2024-04-12T16:10:09.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00ab472f-638f-484a-885d-eab340012a93 -i td/test-erasure-code/1/new.json 2024-04-12T16:10:09.579 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:10:09.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/1/new.json 2024-04-12T16:10:09.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhXBlmzwYHBRAAgzGnug3djzJknN/wLueCaQ== --osd-uuid 00ab472f-638f-484a-885d-eab340012a93 2024-04-12T16:10:09.630 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:09.627+0000 7f1af2fab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:09.632 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:09.631+0000 7f1af2fab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:09.633 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:09.631+0000 7f1af2fab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:09.634 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:09.631+0000 7f1af2fab7c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-04-12T16:10:09.634 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:09.631+0000 7f1af2fab7c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-04-12T16:10:09.634 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:09.631+0000 7f1af2fab7c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-04-12T16:10:09.673 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:09.671+0000 7f1af2fab7c0 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-04-12T16:10:15.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-04-12T16:10:15.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:10:15.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:10:15.667 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:10:15.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:10:16.246 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:10:16.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:10:16.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:10:16.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:10:16.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:10:16.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:10:16.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:16.275+0000 7fe133c297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:16.297 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:16.295+0000 7fe133c297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:16.298 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:16.295+0000 7fe133c297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:10:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:10:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:10:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:10:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:10:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:16.674 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:10:16.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:10:16.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:16.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:16.865 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:16.863+0000 7fe133c297c0 -1 Falling back to public interface 2024-04-12T16:10:17.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:18.077 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:10:18.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:18.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:18.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:10:18.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:18.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:18.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:19.372 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:19.371+0000 7fe133c297c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:10:19.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:19.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:19.555 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:10:19.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:10:19.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:19.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:20.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:21.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:21.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:21.028 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:10:21.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:10:21.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:21.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:21.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:22.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:22.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:22.546 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:10:22.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:10:22.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:22.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:23.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:24.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:24.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:24.113 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:10:24.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:10:24.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:24.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:24.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:25.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:25.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:25.587 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:10:25.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:10:25.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:25.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:26.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:27.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:27.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:27.007 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:10:27.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:10:27.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:27.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:27.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:28.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:28.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:28.570 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:10:28.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:10:28.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:28.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:29.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:30.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:30.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:30.083 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:10:30.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:10:30.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:30.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:30.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:31.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:31.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:31.545 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:10:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:10:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:32.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:33.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:33.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:33.197 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:10:33.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:10:33.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:33.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:34.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:35.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:35.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:35.014 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:10:35.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:10:35.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:35.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:35.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:36.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:36.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:36.981 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:10:36.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:10:36.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:36.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:38.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:39.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:39.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:39.113 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:10:39.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:10:39.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:39.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:39.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:40.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:40.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:40.888 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:10:40.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:10:40.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:40.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:41.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:42.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:42.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:42.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:10:42.697 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:10:42.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:42.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:10:43.170 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/321308787,v1:127.0.0.1:6811/321308787] [v2:127.0.0.1:6812/321308787,v1:127.0.0.1:6813/321308787] exists,up 00ab472f-638f-484a-885d-eab340012a93 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/2 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:10:43.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:10:43.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:10:43.176 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:10:43.176 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:10:43.176 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:10:43.176 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:10:43.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:10:43.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:10:43.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/2 2024-04-12T16:10:43.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:10:43.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8f68c4a-a5a1-485d-899c-56c1542b47cd 2024-04-12T16:10:43.178 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 f8f68c4a-a5a1-485d-899c-56c1542b47cd 2024-04-12T16:10:43.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f8f68c4a-a5a1-485d-899c-56c1542b47cd' 2024-04-12T16:10:43.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:10:43.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADXRlm5qPJCxAAF002FBp1bi6ySMFglvh/+w== 2024-04-12T16:10:43.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADXRlm5qPJCxAAF002FBp1bi6ySMFglvh/+w=="}' 2024-04-12T16:10:43.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8f68c4a-a5a1-485d-899c-56c1542b47cd -i td/test-erasure-code/2/new.json 2024-04-12T16:10:43.663 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:10:43.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/2/new.json 2024-04-12T16:10:43.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADXRlm5qPJCxAAF002FBp1bi6ySMFglvh/+w== --osd-uuid f8f68c4a-a5a1-485d-899c-56c1542b47cd 2024-04-12T16:10:43.712 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:43.711+0000 7f38215997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:43.722 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:43.719+0000 7f38215997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:43.719+0000 7f38215997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:43.723 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:43.723+0000 7f38215997c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-04-12T16:10:43.724 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:43.723+0000 7f38215997c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-04-12T16:10:43.724 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:43.723+0000 7f38215997c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-04-12T16:10:43.782 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:43.779+0000 7f38215997c0 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-04-12T16:10:49.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-04-12T16:10:49.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:10:49.914 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:10:49.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:10:49.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:10:50.469 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:10:50.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:10:50.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:10:50.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:10:50.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:10:50.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:10:50.500 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:50.499+0000 7f109d7017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:50.510 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:50.507+0000 7f109d7017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:50.511 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:50.507+0000 7f109d7017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:10:50.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:10:50.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:10:50.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:10:50.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:10:50.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:10:50.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:50.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:10:50.917 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:10:50.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:50.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:10:51.073 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:51.071+0000 7f109d7017c0 -1 Falling back to public interface 2024-04-12T16:10:51.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:52.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:52.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:52.316 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:10:52.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:10:52.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:52.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:10:52.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:53.610 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:10:53.607+0000 7f109d7017c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:10:53.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:53.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:53.770 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:10:53.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:10:53.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:53.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:10:54.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:55.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:55.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:55.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:10:55.270 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:10:55.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:55.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:10:55.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:56.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:56.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:56.716 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:10:56.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:10:56.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:56.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:10:57.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:10:58.401 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:10:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:10:58.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:10:59.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:10:59.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:10:59.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:10:59.971 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:10:59.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:10:59.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:00.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:01.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:01.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:01.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:11:01.598 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:11:01.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:01.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:02.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:03.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:03.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:03.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:11:03.021 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:11:03.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:03.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:03.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:04.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:04.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:04.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:11:04.515 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:11:04.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:04.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:05.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:06.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:06.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:06.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:11:06.032 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:11:06.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:06.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:07.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:08.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:08.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:08.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:11:08.227 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:11:08.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:08.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:09.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:10.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:10.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:10.070 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:11:10.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:11:10.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:10.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:10.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:11.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:11.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:11.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:11:11.804 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:11:11.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:11.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:12.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:13.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:13.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:13.554 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:11:13.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:11:13.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:13.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:14.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:15.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:15.257 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:11:15.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:15.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:11:15.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:15.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:15.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:16.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:16.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:16.948 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:11:16.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:11:16.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:16.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:11:17.352 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1440940494,v1:127.0.0.1:6819/1440940494] [v2:127.0.0.1:6820/1440940494,v1:127.0.0.1:6821/1440940494] exists,up f8f68c4a-a5a1-485d-899c-56c1542b47cd 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/3 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:11:17.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:11:17.356 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:11:17.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/3 2024-04-12T16:11:17.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:11:17.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9646f915-4a42-42e7-930b-51196c2b31a1 2024-04-12T16:11:17.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9646f915-4a42-42e7-930b-51196c2b31a1' 2024-04-12T16:11:17.359 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 9646f915-4a42-42e7-930b-51196c2b31a1 2024-04-12T16:11:17.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:11:17.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlXRlm5Sd4FhAAXV0eHmtUPPRf1St8N40J+w== 2024-04-12T16:11:17.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlXRlm5Sd4FhAAXV0eHmtUPPRf1St8N40J+w=="}' 2024-04-12T16:11:17.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9646f915-4a42-42e7-930b-51196c2b31a1 -i td/test-erasure-code/3/new.json 2024-04-12T16:11:17.919 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:11:17.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/3/new.json 2024-04-12T16:11:17.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlXRlm5Sd4FhAAXV0eHmtUPPRf1St8N40J+w== --osd-uuid 9646f915-4a42-42e7-930b-51196c2b31a1 2024-04-12T16:11:17.964 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:17.962+0000 7f20941c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:17.978 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:17.974+0000 7f20941c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:17.979 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:17.974+0000 7f20941c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:17.980 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:17.978+0000 7f20941c57c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-04-12T16:11:17.980 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:17.978+0000 7f20941c57c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-04-12T16:11:17.980 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:17.978+0000 7f20941c57c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-04-12T16:11:18.018 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:18.014+0000 7f20941c57c0 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-04-12T16:11:23.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-04-12T16:11:23.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:11:23.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T16:11:23.862 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T16:11:23.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:11:24.414 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:11:24.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T16:11:24.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:11:24.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:11:24.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:11:24.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:11:24.441 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:24.438+0000 7f9397cdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:24.476 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:24.474+0000 7f9397cdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:24.477 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:24.474+0000 7f9397cdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:24.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T16:11:24.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:11:24.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:11:24.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:11:24.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:11:24.850 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:11:24.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:24.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:11:24.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:24.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:25.034 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:25.030+0000 7f9397cdc7c0 -1 Falling back to public interface 2024-04-12T16:11:25.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:26.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:26.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:26.238 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:11:26.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:11:26.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:26.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:26.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:27.531 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:27.526+0000 7f9397cdc7c0 -1 osd.3 0 log_to_monitors true 2024-04-12T16:11:27.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:27.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:27.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:11:27.636 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:11:27.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:27.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:28.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:29.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:11:29.251 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:11:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:29.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:29.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:30.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:30.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:30.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:11:30.665 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:11:30.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:30.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:31.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:32.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:32.090 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:11:32.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:11:32.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:32.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:32.579 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:33.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:33.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:33.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:11:33.581 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:11:33.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:33.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:33.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:34.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:34.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:34.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:11:34.986 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:11:34.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:34.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:35.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:36.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:36.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:36.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:11:36.487 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:11:36.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:36.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:36.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:37.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:37.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:37.972 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:11:37.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:11:37.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:37.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:38.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:39.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:39.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:39.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:11:39.529 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:11:39.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:39.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:40.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:41.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:41.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:41.145 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:11:41.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:11:41.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:41.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:41.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:42.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:42.937 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:11:42.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:42.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:11:42.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:42.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:43.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:44.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:44.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:44.968 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:11:44.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:11:44.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:44.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:45.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:46.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:46.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:46.934 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:11:46.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:11:46.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:46.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:48.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:49.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:49.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:49.015 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:11:49.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:11:49.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:49.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:49.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:11:50.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:11:50.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:50.722 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:11:50.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:11:50.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:50.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:11:51.172 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1383348646,v1:127.0.0.1:6827/1383348646] [v2:127.0.0.1:6828/1383348646,v1:127.0.0.1:6829/1383348646] exists,up 9646f915-4a42-42e7-930b-51196c2b31a1 2024-04-12T16:11:51.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:11:51.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:11:51.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:11:51.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:11:51.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-04-12T16:11:51.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:11:51.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/4 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:11:51.176 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:11:51.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:11:51.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:11:51.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:11:51.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:11:51.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:11:51.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/4 2024-04-12T16:11:51.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:11:51.178 INFO:tasks.workunit.client.0.smithi137.stdout:add osd4 003e83b3-7b85-4504-bc8a-36f045aa8a19 2024-04-12T16:11:51.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=003e83b3-7b85-4504-bc8a-36f045aa8a19 2024-04-12T16:11:51.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 003e83b3-7b85-4504-bc8a-36f045aa8a19' 2024-04-12T16:11:51.179 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:11:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHXRlm3jHcCxAAjHXb14mFRe7AEZKTOiGp0g== 2024-04-12T16:11:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHXRlm3jHcCxAAjHXb14mFRe7AEZKTOiGp0g=="}' 2024-04-12T16:11:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 003e83b3-7b85-4504-bc8a-36f045aa8a19 -i td/test-erasure-code/4/new.json 2024-04-12T16:11:51.649 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:11:51.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/4/new.json 2024-04-12T16:11:51.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHXRlm3jHcCxAAjHXb14mFRe7AEZKTOiGp0g== --osd-uuid 003e83b3-7b85-4504-bc8a-36f045aa8a19 2024-04-12T16:11:51.703 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:51.698+0000 7f02c44917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:51.707 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:51.702+0000 7f02c44917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:51.708 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:51.706+0000 7f02c44917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:51.709 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:51.706+0000 7f02c44917c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-04-12T16:11:51.709 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:51.706+0000 7f02c44917c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-04-12T16:11:51.709 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:51.706+0000 7f02c44917c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-04-12T16:11:51.747 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:51.742+0000 7f02c44917c0 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-04-12T16:11:57.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-04-12T16:11:57.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:11:57.859 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd4 key to auth repository 2024-04-12T16:11:57.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-12T16:11:57.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:11:58.418 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.4 2024-04-12T16:11:58.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-12T16:11:58.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:11:58.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:11:58.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:11:58.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:11:58.443 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:58.438+0000 7fba0b4ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:58.461 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:58.458+0000 7fba0b4ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:58.462 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:58.458+0000 7fba0b4ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:11:58.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-12T16:11:58.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:11:58.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-12T16:11:58.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:11:58.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:11:58.826 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:11:58.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:11:58.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:11:58.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:11:58.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:11:59.022 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:11:59.018+0000 7fba0b4ab7c0 -1 Falling back to public interface 2024-04-12T16:11:59.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:00.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:00.246 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:12:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:12:00.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:00.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:00.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:01.549 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:01.546+0000 7fba0b4ab7c0 -1 osd.4 0 log_to_monitors true 2024-04-12T16:12:01.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:01.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:01.704 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:12:01.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:12:01.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:01.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:02.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:03.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:03.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:03.209 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:12:03.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:12:03.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:03.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:03.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:04.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:04.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:04.651 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:12:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:12:04.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:04.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:05.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:06.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:06.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:06.169 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:12:06.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:12:06.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:06.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:06.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:07.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:07.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:07.762 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:12:07.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:12:07.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:07.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:08.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:09.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:09.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:09.199 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:12:09.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:12:09.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:09.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:09.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:10.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:10.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:10.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:12:10.764 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:12:10.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:10.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:11.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:12.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:12.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:12.406 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:12:12.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:12:12.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:12.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:12.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:13.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:13.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:13.844 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:12:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:12:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:14.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:15.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:15.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:15.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:12:15.289 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:12:15.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:15.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:16.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:17.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:17.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:17.115 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:12:17.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:12:17.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:17.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:18.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:19.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:19.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:19.122 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:12:19.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:12:19.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:19.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:19.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:20.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:20.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:20.884 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:12:20.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:12:20.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:20.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:21.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:22.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:22.940 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:12:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:12:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:22.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:23.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:24.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:24.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:24.719 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:12:24.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:12:24.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:24.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:12:25.127 INFO:tasks.workunit.client.0.smithi137.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/691942262,v1:127.0.0.1:6835/691942262] [v2:127.0.0.1:6836/691942262,v1:127.0.0.1:6837/691942262] exists,up 003e83b3-7b85-4504-bc8a-36f045aa8a19 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:12:25.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/5 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:12:25.131 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:12:25.132 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:12:25.132 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:12:25.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:12:25.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:12:25.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:12:25.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:12:25.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/5 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5d433deb-975f-44d5-87a0-2f491c33be44 2024-04-12T16:12:25.134 INFO:tasks.workunit.client.0.smithi137.stdout:add osd5 5d433deb-975f-44d5-87a0-2f491c33be44 2024-04-12T16:12:25.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 5d433deb-975f-44d5-87a0-2f491c33be44' 2024-04-12T16:12:25.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:12:25.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpXRlmX64eCRAAmrUhbDyjXzQmf+SM4jaH+g== 2024-04-12T16:12:25.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpXRlmX64eCRAAmrUhbDyjXzQmf+SM4jaH+g=="}' 2024-04-12T16:12:25.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5d433deb-975f-44d5-87a0-2f491c33be44 -i td/test-erasure-code/5/new.json 2024-04-12T16:12:25.801 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:12:25.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/5/new.json 2024-04-12T16:12:25.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpXRlmX64eCRAAmrUhbDyjXzQmf+SM4jaH+g== --osd-uuid 5d433deb-975f-44d5-87a0-2f491c33be44 2024-04-12T16:12:25.851 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:25.846+0000 7f5b3902c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:12:25.929 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:25.926+0000 7f5b3902c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:12:25.930 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:25.926+0000 7f5b3902c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:12:25.931 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:25.926+0000 7f5b3902c7c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-04-12T16:12:25.931 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:25.926+0000 7f5b3902c7c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-04-12T16:12:25.931 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:25.926+0000 7f5b3902c7c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-04-12T16:12:25.982 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:25.978+0000 7f5b3902c7c0 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-04-12T16:12:32.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-04-12T16:12:32.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:12:32.103 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd5 key to auth repository 2024-04-12T16:12:32.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-12T16:12:32.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:12:32.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-12T16:12:32.652 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.5 2024-04-12T16:12:32.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:12:32.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:12:32.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:12:32.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:12:32.676 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:32.674+0000 7f336047a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:12:32.679 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:32.674+0000 7f336047a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:12:32.680 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:32.678+0000 7f336047a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:12:33.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-12T16:12:33.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:12:33.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-12T16:12:33.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:12:33.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:12:33.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:33.130 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:12:33.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:12:33.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:33.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:33.266 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:33.262+0000 7f336047a7c0 -1 Falling back to public interface 2024-04-12T16:12:33.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:34.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:34.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:34.614 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:12:34.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:12:34.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:34.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:35.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:35.790 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:12:35.786+0000 7f336047a7c0 -1 osd.5 0 log_to_monitors true 2024-04-12T16:12:36.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:36.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:36.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:12:36.028 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:12:36.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:36.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:36.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:37.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:37.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:37.661 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:12:37.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:12:37.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:37.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:38.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:39.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:39.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:39.089 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:12:39.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:12:39.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:39.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:39.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:40.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:40.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:40.622 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:12:40.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:12:40.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:40.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:41.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:42.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:42.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:42.141 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:12:42.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:12:42.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:42.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:42.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:43.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:43.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:43.603 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:12:43.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:12:43.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:43.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:44.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:45.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:45.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:45.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:12:45.008 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:12:45.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:45.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:45.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:46.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:46.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:46.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:12:46.608 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:12:46.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:46.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:47.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:48.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:48.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:48.119 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:12:48.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:12:48.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:48.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:49.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:50.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:50.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:50.213 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:12:50.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:12:50.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:50.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:51.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:52.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:52.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:52.148 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:12:52.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:12:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:53.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:54.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:54.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:54.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:12:54.307 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:12:54.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:54.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:54.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:55.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:55.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:55.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:12:55.749 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:12:55.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:55.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:56.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:57.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:57.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:57.378 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:12:57.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:12:57.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:57.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:58.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:12:59.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:12:59.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:12:59.397 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:12:59.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:12:59.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:12:59.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:12:59.826 INFO:tasks.workunit.client.0.smithi137.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1656305160,v1:127.0.0.1:6843/1656305160] [v2:127.0.0.1:6844/1656305160,v1:127.0.0.1:6845/1656305160] exists,up 5d433deb-975f-44d5-87a0-2f491c33be44 2024-04-12T16:12:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:12:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:12:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:12:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:12:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-04-12T16:12:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/6 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:12:59.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:12:59.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:12:59.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:12:59.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:12:59.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:12:59.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:12:59.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:12:59.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/6 2024-04-12T16:12:59.835 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:12:59.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc9bb5da-9430-4141-a185-232aec6b8f94 2024-04-12T16:12:59.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 fc9bb5da-9430-4141-a185-232aec6b8f94' 2024-04-12T16:12:59.835 INFO:tasks.workunit.client.0.smithi137.stdout:add osd6 fc9bb5da-9430-4141-a185-232aec6b8f94 2024-04-12T16:12:59.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:12:59.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLXRlmBwytMhAAZxvVdSET1X5izHWQvWC/Fw== 2024-04-12T16:12:59.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLXRlmBwytMhAAZxvVdSET1X5izHWQvWC/Fw=="}' 2024-04-12T16:12:59.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc9bb5da-9430-4141-a185-232aec6b8f94 -i td/test-erasure-code/6/new.json 2024-04-12T16:13:00.315 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:13:00.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/6/new.json 2024-04-12T16:13:00.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLXRlmBwytMhAAZxvVdSET1X5izHWQvWC/Fw== --osd-uuid fc9bb5da-9430-4141-a185-232aec6b8f94 2024-04-12T16:13:00.370 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:00.366+0000 7fd1c1f407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:00.409 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:00.406+0000 7fd1c1f407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:00.410 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:00.406+0000 7fd1c1f407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:00.411 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:00.406+0000 7fd1c1f407c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-04-12T16:13:00.411 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:00.406+0000 7fd1c1f407c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-04-12T16:13:00.411 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:00.406+0000 7fd1c1f407c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-04-12T16:13:00.440 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:00.438+0000 7fd1c1f407c0 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-04-12T16:13:06.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-04-12T16:13:06.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:13:06.331 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd6 key to auth repository 2024-04-12T16:13:06.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-12T16:13:06.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:13:06.887 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.6 2024-04-12T16:13:06.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-12T16:13:06.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:13:06.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:13:06.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:13:06.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:13:06.918 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:06.914+0000 7faabe24a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:06.920 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:06.918+0000 7faabe24a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:06.921 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:06.918+0000 7faabe24a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:07.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-12T16:13:07.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:13:07.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-12T16:13:07.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:13:07.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:13:07.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:07.347 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:13:07.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:13:07.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:07.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:07.453 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:07.450+0000 7faabe24a7c0 -1 Falling back to public interface 2024-04-12T16:13:07.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:08.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:08.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:08.765 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:13:08.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:13:08.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:08.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:09.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:09.983 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:09.978+0000 7faabe24a7c0 -1 osd.6 0 log_to_monitors true 2024-04-12T16:13:10.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:10.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:10.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:13:10.166 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:13:10.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:10.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:10.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:11.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:11.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:11.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:13:11.650 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:13:11.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:11.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:12.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:13.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:13.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:13.103 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:13:13.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:13:13.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:13.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:13.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:14.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:14.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:14.537 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:13:14.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:13:14.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:14.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:15.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:16.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:16.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:16.197 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:13:16.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:13:16.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:16.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:16.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:17.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:17.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:17.776 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:13:17.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:13:17.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:17.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:18.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:19.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:19.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:19.198 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:13:19.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:13:19.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:19.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:19.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:20.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:20.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:20.624 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:13:20.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:13:20.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:20.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:21.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:22.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:22.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:22.229 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:13:22.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:13:22.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:22.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:22.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:23.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:23.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:23.978 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:13:23.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:13:23.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:23.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:24.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:25.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:25.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:25.781 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:13:25.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:13:25.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:25.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:26.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:27.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:27.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:27.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:13:27.540 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:13:27.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:27.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:28.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:29.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:29.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:29.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:13:29.278 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:13:29.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:29.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:30.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:31.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:31.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:31.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:13:31.024 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:13:31.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:31.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:32.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:33.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:33.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:33.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:13:33.081 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:13:33.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:33.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:33.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:34.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:34.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:34.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:13:34.575 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:13:34.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:34.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:13:34.989 INFO:tasks.workunit.client.0.smithi137.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4020703036,v1:127.0.0.1:6851/4020703036] [v2:127.0.0.1:6852/4020703036,v1:127.0.0.1:6853/4020703036] exists,up fc9bb5da-9430-4141-a185-232aec6b8f94 2024-04-12T16:13:34.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/7 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:13:34.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:13:34.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:13:34.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/7 2024-04-12T16:13:34.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:13:34.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=215fad2d-0e2b-4cc9-9573-ce0206d5d99b 2024-04-12T16:13:34.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 215fad2d-0e2b-4cc9-9573-ce0206d5d99b' 2024-04-12T16:13:34.995 INFO:tasks.workunit.client.0.smithi137.stdout:add osd7 215fad2d-0e2b-4cc9-9573-ce0206d5d99b 2024-04-12T16:13:34.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:13:35.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvXRlmf+bXABAARKOAE+nHHPnk2ngrYdPBBQ== 2024-04-12T16:13:35.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvXRlmf+bXABAARKOAE+nHHPnk2ngrYdPBBQ=="}' 2024-04-12T16:13:35.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 215fad2d-0e2b-4cc9-9573-ce0206d5d99b -i td/test-erasure-code/7/new.json 2024-04-12T16:13:35.584 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:13:35.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/7/new.json 2024-04-12T16:13:35.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvXRlmf+bXABAARKOAE+nHHPnk2ngrYdPBBQ== --osd-uuid 215fad2d-0e2b-4cc9-9573-ce0206d5d99b 2024-04-12T16:13:35.636 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:35.634+0000 7ff9aad3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:35.675 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:35.670+0000 7ff9aad3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:35.676 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:35.674+0000 7ff9aad3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:35.677 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:35.674+0000 7ff9aad3e7c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-04-12T16:13:35.677 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:35.674+0000 7ff9aad3e7c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-04-12T16:13:35.677 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:35.674+0000 7ff9aad3e7c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-04-12T16:13:35.705 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:35.702+0000 7ff9aad3e7c0 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-04-12T16:13:42.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-04-12T16:13:42.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:13:42.160 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd7 key to auth repository 2024-04-12T16:13:42.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-04-12T16:13:42.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:13:42.724 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.7 2024-04-12T16:13:42.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-04-12T16:13:42.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:13:42.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:13:42.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:13:42.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:13:42.756 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:42.750+0000 7f8ac5dac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:42.757 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:42.754+0000 7f8ac5dac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:42.759 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:42.754+0000 7f8ac5dac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:13:43.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-04-12T16:13:43.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:13:43.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-04-12T16:13:43.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:13:43.138 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:13:43.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:13:43.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:43.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:13:43.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:43.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:43.337 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:43.334+0000 7f8ac5dac7c0 -1 Falling back to public interface 2024-04-12T16:13:43.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:44.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:44.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:44.553 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:13:44.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:13:44.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:44.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:44.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:45.852 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:13:45.846+0000 7f8ac5dac7c0 -1 osd.7 0 log_to_monitors true 2024-04-12T16:13:45.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:46.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:46.000 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:13:46.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:13:46.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:46.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:46.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:47.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:47.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:47.644 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:13:47.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:13:47.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:47.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:48.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:49.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:49.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:49.089 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:13:49.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:13:49.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:49.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:49.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:50.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:50.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:50.519 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:13:50.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:13:50.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:50.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:51.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:13:52.149 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:13:52.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:52.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:52.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:53.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:53.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:53.762 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:13:53.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:13:53.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:53.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:54.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:55.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:55.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:55.167 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:13:55.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:13:55.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:55.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:55.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:56.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:56.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:56.717 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:13:56.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:13:56.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:56.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:57.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:58.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:58.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:58.129 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:13:58.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:13:58.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:58.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:13:58.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:13:59.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:13:59.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:13:59.747 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:13:59.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:13:59.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:13:59.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:14:00.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:01.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:01.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:01.696 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:14:01.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:14:01.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:01.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:14:02.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:03.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:03.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:03.562 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:14:03.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:14:03.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:03.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:14:04.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:05.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:05.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:05.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:14:05.429 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:14:05.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:05.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:14:06.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:07.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:07.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:07.488 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:14:07.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:14:07.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:07.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:14:08.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:09.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:09.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:09.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:14:09.382 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:14:09.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:09.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:14:09.838 INFO:tasks.workunit.client.0.smithi137.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2250480043,v1:127.0.0.1:6859/2250480043] [v2:127.0.0.1:6860/2250480043,v1:127.0.0.1:6861/2250480043] exists,up 215fad2d-0e2b-4cc9-9573-ce0206d5d99b 2024-04-12T16:14:09.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/8 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:14:09.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:14:09.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:14:09.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/8 2024-04-12T16:14:09.845 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:14:09.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3907bcf1-d922-4aa7-a78c-af118033781e 2024-04-12T16:14:09.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 3907bcf1-d922-4aa7-a78c-af118033781e' 2024-04-12T16:14:09.845 INFO:tasks.workunit.client.0.smithi137.stdout:add osd8 3907bcf1-d922-4aa7-a78c-af118033781e 2024-04-12T16:14:09.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:14:09.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRXRlmBZOUMxAAdauBrxFSTZrlyRNSFiu9cA== 2024-04-12T16:14:09.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRXRlmBZOUMxAAdauBrxFSTZrlyRNSFiu9cA=="}' 2024-04-12T16:14:09.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3907bcf1-d922-4aa7-a78c-af118033781e -i td/test-erasure-code/8/new.json 2024-04-12T16:14:10.495 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:14:10.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/8/new.json 2024-04-12T16:14:10.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRXRlmBZOUMxAAdauBrxFSTZrlyRNSFiu9cA== --osd-uuid 3907bcf1-d922-4aa7-a78c-af118033781e 2024-04-12T16:14:10.549 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:10.546+0000 7f357de747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:10.585 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:10.582+0000 7f357de747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:10.587 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:10.582+0000 7f357de747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:10.587 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:10.582+0000 7f357de747c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-04-12T16:14:10.587 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:10.582+0000 7f357de747c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-04-12T16:14:10.587 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:10.582+0000 7f357de747c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-04-12T16:14:10.693 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:10.690+0000 7f357de747c0 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-04-12T16:14:16.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-04-12T16:14:16.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:14:16.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-04-12T16:14:16.840 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd8 key to auth repository 2024-04-12T16:14:16.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:14:17.414 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.8 2024-04-12T16:14:17.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-04-12T16:14:17.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:14:17.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:14:17.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:14:17.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:14:17.445 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:17.442+0000 7f49d2f1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:17.448 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:17.442+0000 7f49d2f1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:17.449 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:17.446+0000 7f49d2f1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:17.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-04-12T16:14:17.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:14:17.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-04-12T16:14:17.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:14:17.834 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:14:17.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:14:17.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:17.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:14:17.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:17.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:17.994 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:17.990+0000 7f49d2f1c7c0 -1 Falling back to public interface 2024-04-12T16:14:18.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:19.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:19.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:19.262 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:14:19.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:14:19.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:19.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:19.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:20.598 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:20.594+0000 7f49d2f1c7c0 -1 osd.8 0 log_to_monitors true 2024-04-12T16:14:20.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:20.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:20.680 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:14:20.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:14:20.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:20.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:21.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:22.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:22.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:22.263 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:14:22.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:14:22.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:22.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:22.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:23.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:23.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:23.828 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:14:23.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:14:23.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:23.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:24.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:25.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:25.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:25.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:14:25.475 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:14:25.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:25.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:25.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:26.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:26.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:26.926 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:14:26.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:14:26.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:26.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:27.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:28.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:28.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:28.396 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:14:28.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:14:28.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:28.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:28.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:29.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:29.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:29.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:14:29.824 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:14:29.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:29.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:30.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:31.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:31.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:31.462 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:14:31.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:14:31.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:31.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:31.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:32.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:32.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:32.903 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:14:32.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:14:32.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:32.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:33.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:34.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:34.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:34.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:14:34.413 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:14:34.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:34.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:35.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:36.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:36.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:36.423 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:14:36.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:14:36.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:36.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:37.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:38.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:38.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:38.376 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:14:38.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:14:38.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:38.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:39.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:40.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:40.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:40.244 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:14:40.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:14:40.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:40.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:41.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:42.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:42.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:42.166 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:14:42.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:14:42.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:42.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:42.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:43.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:43.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:43.737 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:14:43.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:14:43.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:43.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:44.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:45.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:45.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:45.334 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:14:45.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:14:45.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:45.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:14:45.843 INFO:tasks.workunit.client.0.smithi137.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3167326195,v1:127.0.0.1:6867/3167326195] [v2:127.0.0.1:6868/3167326195,v1:127.0.0.1:6869/3167326195] exists,up 3907bcf1-d922-4aa7-a78c-af118033781e 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:14:45.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/9 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:14:45.846 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:14:45.847 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:14:45.847 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:14:45.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:14:45.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:14:45.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:14:45.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/9 2024-04-12T16:14:45.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:14:45.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd763060-03da-4f04-b7aa-a44a886037c6 2024-04-12T16:14:45.849 INFO:tasks.workunit.client.0.smithi137.stdout:add osd9 bd763060-03da-4f04-b7aa-a44a886037c6 2024-04-12T16:14:45.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 bd763060-03da-4f04-b7aa-a44a886037c6' 2024-04-12T16:14:45.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:14:45.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1XRlm+dnRMxAAkcI13zm5r8QWc95LDe2GpQ== 2024-04-12T16:14:45.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1XRlm+dnRMxAAkcI13zm5r8QWc95LDe2GpQ=="}' 2024-04-12T16:14:45.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd763060-03da-4f04-b7aa-a44a886037c6 -i td/test-erasure-code/9/new.json 2024-04-12T16:14:46.369 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:14:46.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/9/new.json 2024-04-12T16:14:46.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1XRlm+dnRMxAAkcI13zm5r8QWc95LDe2GpQ== --osd-uuid bd763060-03da-4f04-b7aa-a44a886037c6 2024-04-12T16:14:46.414 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:46.410+0000 7fab38dd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:46.486 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:46.482+0000 7fab38dd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:46.488 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:46.482+0000 7fab38dd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:46.488 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:46.486+0000 7fab38dd57c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-04-12T16:14:46.488 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:46.486+0000 7fab38dd57c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-04-12T16:14:46.488 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:46.486+0000 7fab38dd57c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-04-12T16:14:46.618 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:46.614+0000 7fab38dd57c0 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-04-12T16:14:53.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-04-12T16:14:53.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:14:53.738 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd9 key to auth repository 2024-04-12T16:14:53.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-04-12T16:14:53.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:14:54.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-04-12T16:14:54.298 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.9 2024-04-12T16:14:54.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:14:54.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:14:54.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:14:54.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:14:54.322 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:54.318+0000 7f826ac887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:54.357 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:54.354+0000 7f826ac887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:54.358 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:54.354+0000 7f826ac887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:14:54.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-04-12T16:14:54.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:14:54.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-12T16:14:54.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:14:54.744 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:14:54.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:14:54.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:54.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:14:54.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:54.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:14:54.950 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:54.946+0000 7f826ac887c0 -1 Falling back to public interface 2024-04-12T16:14:55.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:56.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:56.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:56.186 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:14:56.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:14:56.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:56.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:14:56.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:57.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:57.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:57.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:14:57.596 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:14:57.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:14:57.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:57.597 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:14:57.594+0000 7f826ac887c0 -1 osd.9 0 log_to_monitors true 2024-04-12T16:14:58.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:14:59.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:14:59.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:14:59.192 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:14:59.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:14:59.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:14:59.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:14:59.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:00.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:00.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:00.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:15:00.781 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:15:00.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:00.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:01.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:02.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:02.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:02.338 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:15:02.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:15:02.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:02.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:03.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:04.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:04.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:04.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:15:04.069 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:15:04.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:04.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:04.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:05.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:05.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:05.758 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:15:05.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:15:05.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:05.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:06.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:07.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:07.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:07.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:15:07.198 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:15:07.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:07.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:07.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:08.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:08.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:08.646 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:15:08.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:15:08.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:08.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:09.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:10.143 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:15:10.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:15:10.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:10.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:10.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:11.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:11.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:11.797 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:15:11.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:15:11.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:11.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:12.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:13.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:13.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:13.333 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:15:13.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:15:13.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:13.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:14.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:15.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:15.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:15.180 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:15:15.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:15:15.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:15.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:16.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:17.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:17.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:17.231 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:15:17.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:15:17.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:17.231 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:18.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:19.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:19.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:19.155 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:15:19.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:15:19.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:19.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:20.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:21.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:21.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:21.143 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:15:21.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:15:21.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:21.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:22.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:23.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:23.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:23.292 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:15:23.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:15:23.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:23.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:23.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:24.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:24.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:24.783 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:15:24.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:15:24.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:24.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:15:25.212 INFO:tasks.workunit.client.0.smithi137.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2606182997,v1:127.0.0.1:6875/2606182997] [v2:127.0.0.1:6876/2606182997,v1:127.0.0.1:6877/2606182997] exists,up bd763060-03da-4f04-b7aa-a44a886037c6 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=10 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/10 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:15:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:15:25.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:15:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/10 2024-04-12T16:15:25.219 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:15:25.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a3335886-7181-4f6d-8c2c-bfac4e02afac 2024-04-12T16:15:25.219 INFO:tasks.workunit.client.0.smithi137.stdout:add osd10 a3335886-7181-4f6d-8c2c-bfac4e02afac 2024-04-12T16:15:25.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd10 a3335886-7181-4f6d-8c2c-bfac4e02afac' 2024-04-12T16:15:25.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:15:25.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdXhlmNyUyDhAACc6TELDa1uG1rp2QLGWW2g== 2024-04-12T16:15:25.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdXhlmNyUyDhAACc6TELDa1uG1rp2QLGWW2g=="}' 2024-04-12T16:15:25.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a3335886-7181-4f6d-8c2c-bfac4e02afac -i td/test-erasure-code/10/new.json 2024-04-12T16:15:25.713 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:15:25.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/10/new.json 2024-04-12T16:15:25.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 10 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdXhlmNyUyDhAACc6TELDa1uG1rp2QLGWW2g== --osd-uuid a3335886-7181-4f6d-8c2c-bfac4e02afac 2024-04-12T16:15:25.763 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:25.758+0000 7f300eb387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:15:25.817 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:25.814+0000 7f300eb387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:15:25.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:25.814+0000 7f300eb387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:15:25.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:25.814+0000 7f300eb387c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-04-12T16:15:25.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:25.814+0000 7f300eb387c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-04-12T16:15:25.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:25.814+0000 7f300eb387c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-04-12T16:15:25.941 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:25.938+0000 7f300eb387c0 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-04-12T16:15:32.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-04-12T16:15:32.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:15:32.958 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd10 key to auth repository 2024-04-12T16:15:32.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd10 key to auth repository 2024-04-12T16:15:32.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:15:33.515 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.10 2024-04-12T16:15:33.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.10 2024-04-12T16:15:33.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 10 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:15:33.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:15:33.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:15:33.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:15:33.546 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:33.542+0000 7ff20e9257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:15:33.548 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:33.546+0000 7ff20e9257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:15:33.550 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:33.546+0000 7ff20e9257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:15:33.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 10 2024-04-12T16:15:33.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:15:33.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=10 2024-04-12T16:15:33.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:15:33.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:15:33.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:33.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:15:33.908 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:15:33.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:33.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:34.097 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:34.094+0000 7ff20e9257c0 -1 Falling back to public interface 2024-04-12T16:15:34.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:35.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:35.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:35.428 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:15:35.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:15:35.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:35.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:35.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:36.839 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:15:36.834+0000 7ff20e9257c0 -1 osd.10 0 log_to_monitors true 2024-04-12T16:15:36.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:36.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:36.841 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:15:36.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:15:36.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:36.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:37.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:38.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:38.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:38.434 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:15:38.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:15:38.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:38.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:38.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:39.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:39.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:39.954 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:15:39.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:15:39.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:39.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:40.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:41.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:41.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:41.506 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:15:41.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:15:41.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:41.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:41.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:42.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:42.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:42.950 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:15:42.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:15:42.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:42.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:43.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:44.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:44.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:44.409 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:15:44.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:15:44.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:44.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:44.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:45.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:45.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:45.913 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:15:45.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:15:45.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:45.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:46.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:47.354 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:15:47.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:15:47.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:47.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:47.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:48.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:48.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:48.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:15:48.881 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:15:48.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:48.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:49.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:50.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:50.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:50.325 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:15:50.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:15:50.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:50.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:50.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:51.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:51.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:51.790 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:15:51.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:15:51.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:51.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:52.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:53.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:53.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:53.671 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:15:53.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:15:53.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:53.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:54.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:55.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:55.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:55.479 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:15:55.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:15:55.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:55.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:56.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:57.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:57.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:57.393 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:15:57.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:15:57.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:57.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:58.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:15:59.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:15:59.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:15:59.269 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:15:59.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:15:59.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:15:59.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:15:59.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:16:00.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:16:00.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:00.932 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:16:00.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:16:00.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:00.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:16:01.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:16:02.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:16:02.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:02.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:16:02.945 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:16:02.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:02.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-04-12T16:16:03.392 INFO:tasks.workunit.client.0.smithi137.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/934290214,v1:127.0.0.1:6883/934290214] [v2:127.0.0.1:6884/934290214,v1:127.0.0.1:6885/934290214] exists,up a3335886-7181-4f6d-8c2c-bfac4e02afac 2024-04-12T16:16:03.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:16:03.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:16:03.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:16:03.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-04-12T16:16:03.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-12T16:16:03.790 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' does not exist 2024-04-12T16:16:03.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-12T16:16:03.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:16:04.468 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:16:04.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:16:05.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-12T16:16:09.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-04-12T16:16:09.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:16:09.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:16:09.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:16:09.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:16:09.523 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:16:09.523 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:16:09.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:16:09.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:16:09.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:16:09.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:16:09.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:16:09.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:16:09.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:16:09.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:16:09.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:16:10.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:16:10.141 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:16:10.141 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:10.142 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:16:10.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836558 2024-04-12T16:16:10.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836558 2024-04-12T16:16:10.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558' 2024-04-12T16:16:10.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:10.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:16:10.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-04-12T16:16:10.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-04-12T16:16:10.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035' 2024-04-12T16:16:10.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:10.540 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:16:10.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-04-12T16:16:10.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-04-12T16:16:10.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506' 2024-04-12T16:16:10.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:10.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:16:10.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345980 2024-04-12T16:16:10.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345980 2024-04-12T16:16:10.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506 3-85899345980' 2024-04-12T16:16:10.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:10.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:16:11.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182452 2024-04-12T16:16:11.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182452 2024-04-12T16:16:11.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506 3-85899345980 4-107374182452' 2024-04-12T16:16:11.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:11.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:16:11.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018927 2024-04-12T16:16:11.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018927 2024-04-12T16:16:11.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506 3-85899345980 4-107374182452 5-128849018927' 2024-04-12T16:16:11.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:11.219 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:16:11.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855394 2024-04-12T16:16:11.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855394 2024-04-12T16:16:11.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506 3-85899345980 4-107374182452 5-128849018927 6-150323855394' 2024-04-12T16:16:11.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:11.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:16:11.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691874 2024-04-12T16:16:11.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691874 2024-04-12T16:16:11.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506 3-85899345980 4-107374182452 5-128849018927 6-150323855394 7-171798691874' 2024-04-12T16:16:11.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:11.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:16:11.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528347 2024-04-12T16:16:11.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528347 2024-04-12T16:16:11.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506 3-85899345980 4-107374182452 5-128849018927 6-150323855394 7-171798691874 8-193273528347' 2024-04-12T16:16:11.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:11.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:16:11.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364816 2024-04-12T16:16:11.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364816 2024-04-12T16:16:11.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506 3-85899345980 4-107374182452 5-128849018927 6-150323855394 7-171798691874 8-193273528347 9-214748364816' 2024-04-12T16:16:11.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:11.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:16:11.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201285 2024-04-12T16:16:11.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201285 2024-04-12T16:16:11.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509506 3-85899345980 4-107374182452 5-128849018927 6-150323855394 7-171798691874 8-193273528347 9-214748364816 10-236223201285' 2024-04-12T16:16:11.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:11.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836558 2024-04-12T16:16:11.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:11.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:16:11.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836558 2024-04-12T16:16:11.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:11.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836558 2024-04-12T16:16:11.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-04-12T16:16:11.966 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836558 2024-04-12T16:16:11.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:16:12.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836558 2024-04-12T16:16:12.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T16:16:13.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T16:16:13.352 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:16:13.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836558 2024-04-12T16:16:13.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:13.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:13.755 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-04-12T16:16:13.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:16:13.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-04-12T16:16:13.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:13.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-04-12T16:16:13.758 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949673035 2024-04-12T16:16:13.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-04-12T16:16:13.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:16:14.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673035 -lt 42949673035 2024-04-12T16:16:14.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:14.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-04-12T16:16:14.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:14.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:16:14.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-04-12T16:16:14.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:14.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-04-12T16:16:14.170 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509506 2024-04-12T16:16:14.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-04-12T16:16:14.171 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:16:14.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509506 2024-04-12T16:16:14.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:14.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345980 2024-04-12T16:16:14.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:14.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:16:14.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345980 2024-04-12T16:16:14.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:14.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345980 2024-04-12T16:16:14.625 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 85899345980 2024-04-12T16:16:14.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345980' 2024-04-12T16:16:14.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:16:15.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345981 -lt 85899345980 2024-04-12T16:16:15.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:15.112 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182452 2024-04-12T16:16:15.112 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:15.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:16:15.113 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182452 2024-04-12T16:16:15.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:15.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182452 2024-04-12T16:16:15.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-04-12T16:16:15.115 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 107374182452 2024-04-12T16:16:15.116 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:16:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182452 -lt 107374182452 2024-04-12T16:16:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:15.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:15.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018927 2024-04-12T16:16:15.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:16:15.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018927 2024-04-12T16:16:15.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:15.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018927 2024-04-12T16:16:15.610 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 128849018927 2024-04-12T16:16:15.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018927' 2024-04-12T16:16:15.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:16:16.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018927 -lt 128849018927 2024-04-12T16:16:16.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:16.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855394 2024-04-12T16:16:16.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:16.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:16:16.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855394 2024-04-12T16:16:16.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855394 2024-04-12T16:16:16.260 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 150323855394 2024-04-12T16:16:16.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-04-12T16:16:16.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:16:16.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855395 -lt 150323855394 2024-04-12T16:16:16.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:16.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:16.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691874 2024-04-12T16:16:16.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:16:16.676 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691874 2024-04-12T16:16:16.676 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:16.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691874 2024-04-12T16:16:16.677 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.7 seq 171798691874 2024-04-12T16:16:16.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-04-12T16:16:16.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:16:17.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691875 -lt 171798691874 2024-04-12T16:16:17.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:17.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528347 2024-04-12T16:16:17.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:17.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:16:17.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528347 2024-04-12T16:16:17.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:17.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528347 2024-04-12T16:16:17.089 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.8 seq 193273528347 2024-04-12T16:16:17.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2024-04-12T16:16:17.089 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:16:17.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528348 -lt 193273528347 2024-04-12T16:16:17.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:17.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364816 2024-04-12T16:16:17.524 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:17.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:16:17.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364816 2024-04-12T16:16:17.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:17.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364816 2024-04-12T16:16:17.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364816' 2024-04-12T16:16:17.528 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.9 seq 214748364816 2024-04-12T16:16:17.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:16:18.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364817 -lt 214748364816 2024-04-12T16:16:18.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:18.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201285 2024-04-12T16:16:18.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:18.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:16:18.036 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201285 2024-04-12T16:16:18.036 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:18.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201285 2024-04-12T16:16:18.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201285' 2024-04-12T16:16:18.038 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.10 seq 236223201285 2024-04-12T16:16:18.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:16:18.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201286 -lt 236223201285 2024-04-12T16:16:18.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:16:18.480 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:16:18.480 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:16:19.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-12T16:16:19.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:16:19.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:16:19.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:16:19.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:16:19.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:16:19.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:16:19.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:16:19.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T16:16:19.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:16:19.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:16:19.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:16:20.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-12T16:16:20.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:16:20.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:16:20.029 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-04-12T16:16:20.029 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:16:20.030 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:16:20.030 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:16:20.030 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:16:20.030 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:16:20.030 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48107/ceph-osd.0.asok 2024-04-12T16:16:20.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-04-12T16:16:20.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.48107/ceph-osd.0.asok log flush 2024-04-12T16:16:20.136 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:16:20.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-04-12T16:16:20.154 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:09:43.331+0000 7f02e54fe7c0 0 load: jerasure load: lrc 2024-04-12T16:16:20.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-04-12T16:16:20.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-04-12T16:16:20.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-04-12T16:16:20.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-04-12T16:16:20.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-04-12T16:16:21.577 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'ecpool' already exists 2024-04-12T16:16:21.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:16:22.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-04-12T16:16:22.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:16:22.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:16:22.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:16:22.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:16:22.599 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:16:22.599 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:16:22.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:16:22.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:16:22.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:16:22.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:16:22.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:16:22.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:16:22.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:16:22.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:16:22.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:16:23.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:16:23.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:23.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:16:23.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836563 2024-04-12T16:16:23.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836563 2024-04-12T16:16:23.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563' 2024-04-12T16:16:23.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:23.827 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:16:24.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-04-12T16:16:24.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-04-12T16:16:24.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040' 2024-04-12T16:16:24.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:24.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:16:24.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-04-12T16:16:24.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-04-12T16:16:24.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511' 2024-04-12T16:16:24.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:24.673 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:16:24.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345985 2024-04-12T16:16:24.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345985 2024-04-12T16:16:24.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511 3-85899345985' 2024-04-12T16:16:24.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:24.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:16:25.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182457 2024-04-12T16:16:25.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182457 2024-04-12T16:16:25.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511 3-85899345985 4-107374182457' 2024-04-12T16:16:25.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:25.309 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:16:25.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018932 2024-04-12T16:16:25.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018932 2024-04-12T16:16:25.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511 3-85899345985 4-107374182457 5-128849018932' 2024-04-12T16:16:25.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:25.541 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:16:25.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855399 2024-04-12T16:16:25.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855399 2024-04-12T16:16:25.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511 3-85899345985 4-107374182457 5-128849018932 6-150323855399' 2024-04-12T16:16:25.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:25.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:16:25.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691879 2024-04-12T16:16:25.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691879 2024-04-12T16:16:25.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511 3-85899345985 4-107374182457 5-128849018932 6-150323855399 7-171798691879' 2024-04-12T16:16:25.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:25.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:16:26.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528352 2024-04-12T16:16:26.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528352 2024-04-12T16:16:26.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511 3-85899345985 4-107374182457 5-128849018932 6-150323855399 7-171798691879 8-193273528352' 2024-04-12T16:16:26.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:26.139 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:16:26.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364821 2024-04-12T16:16:26.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364821 2024-04-12T16:16:26.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511 3-85899345985 4-107374182457 5-128849018932 6-150323855399 7-171798691879 8-193273528352 9-214748364821' 2024-04-12T16:16:26.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:26.307 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:16:26.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201290 2024-04-12T16:16:26.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201290 2024-04-12T16:16:26.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509511 3-85899345985 4-107374182457 5-128849018932 6-150323855399 7-171798691879 8-193273528352 9-214748364821 10-236223201290' 2024-04-12T16:16:26.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:26.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:26.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836563 2024-04-12T16:16:26.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:16:26.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836563 2024-04-12T16:16:26.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:26.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836563 2024-04-12T16:16:26.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-04-12T16:16:26.468 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836563 2024-04-12T16:16:26.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:16:26.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836563 2024-04-12T16:16:26.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:26.872 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-04-12T16:16:26.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:26.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:16:26.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-04-12T16:16:26.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:26.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-04-12T16:16:26.877 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949673040 2024-04-12T16:16:26.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-04-12T16:16:26.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:16:27.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673040 2024-04-12T16:16:27.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:27.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-04-12T16:16:27.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:27.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:16:27.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-04-12T16:16:27.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:27.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-04-12T16:16:27.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-04-12T16:16:27.375 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509511 2024-04-12T16:16:27.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:16:27.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509511 -lt 64424509511 2024-04-12T16:16:27.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:27.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:27.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345985 2024-04-12T16:16:27.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:16:27.853 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345985 2024-04-12T16:16:27.854 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:27.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345985 2024-04-12T16:16:27.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345985' 2024-04-12T16:16:27.855 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 85899345985 2024-04-12T16:16:27.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:16:28.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345986 -lt 85899345985 2024-04-12T16:16:28.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:28.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182457 2024-04-12T16:16:28.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:28.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:16:28.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182457 2024-04-12T16:16:28.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:28.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182457 2024-04-12T16:16:28.317 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 107374182457 2024-04-12T16:16:28.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182457' 2024-04-12T16:16:28.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:16:28.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182457 -lt 107374182457 2024-04-12T16:16:28.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:28.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018932 2024-04-12T16:16:28.738 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:28.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:16:28.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018932 2024-04-12T16:16:28.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:28.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018932 2024-04-12T16:16:28.742 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 128849018932 2024-04-12T16:16:28.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018932' 2024-04-12T16:16:28.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:16:29.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018932 -lt 128849018932 2024-04-12T16:16:29.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:29.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855399 2024-04-12T16:16:29.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:29.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:16:29.190 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855399 2024-04-12T16:16:29.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:29.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855399 2024-04-12T16:16:29.192 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 150323855399 2024-04-12T16:16:29.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-04-12T16:16:29.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:16:29.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855399 -lt 150323855399 2024-04-12T16:16:29.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:29.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691879 2024-04-12T16:16:29.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:29.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:16:29.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691879 2024-04-12T16:16:29.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:29.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691879 2024-04-12T16:16:29.620 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.7 seq 171798691879 2024-04-12T16:16:29.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-04-12T16:16:29.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:16:30.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691879 2024-04-12T16:16:30.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:30.068 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:30.068 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528352 2024-04-12T16:16:30.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:16:30.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528352 2024-04-12T16:16:30.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:30.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528352 2024-04-12T16:16:30.072 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.8 seq 193273528352 2024-04-12T16:16:30.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2024-04-12T16:16:30.072 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:16:30.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528352 -lt 193273528352 2024-04-12T16:16:30.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:30.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364821 2024-04-12T16:16:30.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:30.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:16:30.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364821 2024-04-12T16:16:30.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:30.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364821 2024-04-12T16:16:30.494 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.9 seq 214748364821 2024-04-12T16:16:30.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364821' 2024-04-12T16:16:30.495 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:16:30.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364822 -lt 214748364821 2024-04-12T16:16:30.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:30.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201290 2024-04-12T16:16:30.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:30.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:16:30.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201290 2024-04-12T16:16:30.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:30.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201290 2024-04-12T16:16:30.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201290' 2024-04-12T16:16:30.970 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.10 seq 236223201290 2024-04-12T16:16:30.970 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:16:31.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201291 -lt 236223201290 2024-04-12T16:16:31.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:16:31.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:16:31.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:16:31.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-12T16:16:31.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:16:31.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:16:31.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:16:31.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:16:31.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:16:31.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:16:31.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:16:32.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-12T16:16:32.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:16:32.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:16:32.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:16:32.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-12T16:16:32.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:16:32.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:16:32.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-04-12T16:16:32.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-12T16:16:32.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-04-12T16:16:32.885 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_chunk_mapping 2024-04-12T16:16:32.885 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_put_get_isa 2024-04-12T16:16:32.885 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_put_get_jerasure 2024-04-12T16:16:32.885 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_put_get_lrc_advanced 2024-04-12T16:16:32.886 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_put_get_lrc_kml 2024-04-12T16:16:32.886 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_put_get_shec' 2024-04-12T16:16:32.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-04-12T16:16:32.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-04-12T16:16:32.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2024-04-12T16:16:32.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2024-04-12T16:16:32.886 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-04-12T16:16:32.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2024-04-12T16:16:32.902 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-04-12T16:16:32.902 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2024-04-12T16:16:33.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2024-04-12T16:16:33.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2024-04-12T16:16:33.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2024-04-12T16:16:33.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-04-12T16:16:33.307 INFO:tasks.workunit.client.0.smithi137.stderr:2+0 records in 2024-04-12T16:16:33.307 INFO:tasks.workunit.client.0.smithi137.stderr:2+0 records out 2024-04-12T16:16:33.307 INFO:tasks.workunit.client.0.smithi137.stderr:16382 bytes (16 kB, 16 KiB) copied, 0.000167772 s, 97.6 MB/s 2024-04-12T16:16:33.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-04-12T16:16:33.356 INFO:tasks.workunit.client.0.smithi137.stderr:INFO: op_size has been rounded to 8192 2024-04-12T16:16:33.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-04-12T16:16:33.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-04-12T16:16:33.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-04-12T16:16:33.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-04-12T16:16:33.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-04-12T16:16:33.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-04-12T16:16:33.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2024-04-12T16:16:33.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2024-04-12T16:16:33.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2024-04-12T16:16:33.803 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-04-12T16:16:33.803 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-04-12T16:16:33.824 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-04-12T16:16:33.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-04-12T16:16:33.824 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-04-12T16:16:33.825 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-04-12T16:16:33.846 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-04-12T16:16:33.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-04-12T16:16:33.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-04-12T16:16:33.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-04-12T16:16:33.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-04-12T16:16:34.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-04-12T16:16:34.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-04-12T16:16:34.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-12T16:16:34.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-04-12T16:16:34.169 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-04-12T16:16:34.169 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:16:34.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr:0' 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-04-12T16:16:34.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-04-12T16:16:34.758 INFO:tasks.workunit.client.0.smithi137.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-12T16:16:34.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-04-12T16:16:34.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-04-12T16:16:34.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-04-12T16:16:34.886 INFO:tasks.workunit.client.0.smithi137.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-12T16:16:34.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-04-12T16:16:34.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-04-12T16:16:34.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-04-12T16:16:35.016 INFO:tasks.workunit.client.0.smithi137.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-12T16:16:35.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-04-12T16:16:35.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-04-12T16:16:35.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-04-12T16:16:35.145 INFO:tasks.workunit.client.0.smithi137.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-12T16:16:35.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-04-12T16:16:35.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-04-12T16:16:35.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:16:35.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-04-12T16:16:35.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-04-12T16:16:35.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-12T16:16:35.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-04-12T16:16:35.160 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-04-12T16:16:35.161 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr:0' 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-04-12T16:16:35.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:16:35.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:16:35.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:16:35.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-04-12T16:16:35.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-04-12T16:16:35.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T16:16:35.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-12T16:16:35.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T16:16:35.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-04-12T16:16:35.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-04-12T16:16:36.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-04-12T16:16:36.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-04-12T16:16:36.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/3 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T16:16:36.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:16:36.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T16:16:36.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:16:36.437 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:16:36.437 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:16:36.437 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:16:36.437 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:16:36.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/3 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-12T16:16:36.439 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.3 2024-04-12T16:16:36.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:16:36.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/3/whoami 2024-04-12T16:16:36.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-12T16:16:36.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T16:16:36.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T16:16:36.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T16:16:36.475 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:36.474+0000 7fb77911b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:16:36.484 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:36.482+0000 7fb77911b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:16:36.485 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:36.482+0000 7fb77911b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:36.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:16:37.033 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:37.030+0000 7fb77911b7c0 -1 Falling back to public interface 2024-04-12T16:16:37.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:16:38.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:16:38.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:38.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:16:38.445 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:16:38.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:38.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:16:38.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:16:39.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:16:39.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:39.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:16:39.949 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:16:39.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:39.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:16:39.952 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:39.950+0000 7fb77911b7c0 -1 osd.3 77 log_to_monitors true 2024-04-12T16:16:40.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:16:41.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:16:41.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:41.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:16:41.423 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:16:41.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:41.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:16:42.283 INFO:tasks.workunit.client.0.smithi137.stderr:osd.3 up in weight 1 up_from 81 up_thru 81 down_at 78 last_clean_interval [20,77) [v2:127.0.0.1:6826/262548867,v1:127.0.0.1:6827/262548867] [v2:127.0.0.1:6828/262548867,v1:127.0.0.1:6829/262548867] exists,up 9646f915-4a42-42e7-930b-51196c2b31a1 2024-04-12T16:16:42.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:16:42.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:16:42.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:16:42.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T16:16:42.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:16:42.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:16:42.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:16:42.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:16:42.284 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:16:42.285 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:16:42.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:16:42.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:16:42.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:16:42.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:16:42.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:16:42.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:16:42.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:16:42.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:16:42.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:16:42.903 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:16:42.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:16:42.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:42.904 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:16:43.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-04-12T16:16:43.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-04-12T16:16:43.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-04-12T16:16:43.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:43.091 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:16:43.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-04-12T16:16:43.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-04-12T16:16:43.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046' 2024-04-12T16:16:43.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:43.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:16:43.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-04-12T16:16:43.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-04-12T16:16:43.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517' 2024-04-12T16:16:43.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:43.400 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:16:43.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350979 2024-04-12T16:16:43.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350979 2024-04-12T16:16:43.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517 3-347892350979' 2024-04-12T16:16:43.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:43.545 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:16:43.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182462 2024-04-12T16:16:43.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182462 2024-04-12T16:16:43.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517 3-347892350979 4-107374182462' 2024-04-12T16:16:43.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:43.688 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:16:43.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018937 2024-04-12T16:16:43.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018937 2024-04-12T16:16:43.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517 3-347892350979 4-107374182462 5-128849018937' 2024-04-12T16:16:43.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:43.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:16:43.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855404 2024-04-12T16:16:43.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855404 2024-04-12T16:16:43.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517 3-347892350979 4-107374182462 5-128849018937 6-150323855404' 2024-04-12T16:16:43.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:43.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:16:44.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691884 2024-04-12T16:16:44.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691884 2024-04-12T16:16:44.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517 3-347892350979 4-107374182462 5-128849018937 6-150323855404 7-171798691884' 2024-04-12T16:16:44.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:44.113 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:16:44.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528357 2024-04-12T16:16:44.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528357 2024-04-12T16:16:44.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517 3-347892350979 4-107374182462 5-128849018937 6-150323855404 7-171798691884 8-193273528357' 2024-04-12T16:16:44.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:44.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:16:44.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364827 2024-04-12T16:16:44.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364827 2024-04-12T16:16:44.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517 3-347892350979 4-107374182462 5-128849018937 6-150323855404 7-171798691884 8-193273528357 9-214748364827' 2024-04-12T16:16:44.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:44.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:16:44.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201296 2024-04-12T16:16:44.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201296 2024-04-12T16:16:44.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509517 3-347892350979 4-107374182462 5-128849018937 6-150323855404 7-171798691884 8-193273528357 9-214748364827 10-236223201296' 2024-04-12T16:16:44.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:44.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-04-12T16:16:44.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:44.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:16:44.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-04-12T16:16:44.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:44.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-04-12T16:16:44.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-04-12T16:16:44.569 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 21474836568 2024-04-12T16:16:44.569 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:16:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836568 2024-04-12T16:16:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:45.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-04-12T16:16:45.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:45.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:16:45.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-04-12T16:16:45.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:45.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-04-12T16:16:45.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-04-12T16:16:45.067 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 42949673046 2024-04-12T16:16:45.068 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:16:45.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673046 2024-04-12T16:16:45.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:45.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-04-12T16:16:45.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:45.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:16:45.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-04-12T16:16:45.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:45.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-04-12T16:16:45.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-04-12T16:16:45.527 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 64424509517 2024-04-12T16:16:45.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:16:45.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509517 -lt 64424509517 2024-04-12T16:16:45.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:45.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-347892350979 2024-04-12T16:16:45.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:45.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:16:45.956 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-347892350979 2024-04-12T16:16:45.957 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:45.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350979 2024-04-12T16:16:45.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 347892350979' 2024-04-12T16:16:45.958 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 347892350979 2024-04-12T16:16:45.959 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:16:46.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350980 -lt 347892350979 2024-04-12T16:16:46.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:46.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182462 2024-04-12T16:16:46.381 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:46.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:16:46.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:46.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182462 2024-04-12T16:16:46.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182462 2024-04-12T16:16:46.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-04-12T16:16:46.384 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.4 seq 107374182462 2024-04-12T16:16:46.384 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:16:46.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182462 2024-04-12T16:16:46.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:46.790 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:46.790 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018937 2024-04-12T16:16:46.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:16:46.792 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018937 2024-04-12T16:16:46.792 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:46.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018937 2024-04-12T16:16:46.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018937' 2024-04-12T16:16:46.793 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.5 seq 128849018937 2024-04-12T16:16:46.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:16:47.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018938 -lt 128849018937 2024-04-12T16:16:47.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:47.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855404 2024-04-12T16:16:47.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:47.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:16:47.196 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855404 2024-04-12T16:16:47.196 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:47.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855404 2024-04-12T16:16:47.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855404' 2024-04-12T16:16:47.198 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.6 seq 150323855404 2024-04-12T16:16:47.198 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:16:47.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855405 -lt 150323855404 2024-04-12T16:16:47.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:47.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691884 2024-04-12T16:16:47.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:47.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:16:47.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691884 2024-04-12T16:16:47.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:47.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691884 2024-04-12T16:16:47.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-04-12T16:16:47.630 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.7 seq 171798691884 2024-04-12T16:16:47.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:16:48.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691885 -lt 171798691884 2024-04-12T16:16:48.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:48.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528357 2024-04-12T16:16:48.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:48.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:16:48.078 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528357 2024-04-12T16:16:48.079 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:48.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528357 2024-04-12T16:16:48.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-04-12T16:16:48.080 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.8 seq 193273528357 2024-04-12T16:16:48.080 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:16:48.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528358 -lt 193273528357 2024-04-12T16:16:48.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:48.551 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364827 2024-04-12T16:16:48.551 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:48.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:16:48.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364827 2024-04-12T16:16:48.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:48.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364827 2024-04-12T16:16:48.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364827' 2024-04-12T16:16:48.554 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.9 seq 214748364827 2024-04-12T16:16:48.555 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:16:49.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364827 -lt 214748364827 2024-04-12T16:16:49.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:16:49.002 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201296 2024-04-12T16:16:49.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:16:49.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:16:49.005 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201296 2024-04-12T16:16:49.005 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:16:49.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201296 2024-04-12T16:16:49.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201296' 2024-04-12T16:16:49.007 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.10 seq 236223201296 2024-04-12T16:16:49.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:16:49.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201296 -lt 236223201296 2024-04-12T16:16:49.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:16:49.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:16:49.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:16:50.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-12T16:16:50.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:16:50.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:16:50.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:16:50.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:16:50.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:16:50.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:16:50.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:16:50.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-12T16:16:50.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:16:50.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:16:50.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:16:51.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-12T16:16:51.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:16:51.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:16:51.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:16:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:16:51.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:16:51.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:16:51.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:16:51.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-04-12T16:16:51.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-04-12T16:16:51.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T16:16:51.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-12T16:16:51.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T16:16:51.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-04-12T16:16:51.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-04-12T16:16:52.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-04-12T16:16:52.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-04-12T16:16:52.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/5 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T16:16:52.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:16:52.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:16:52.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:16:52.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:16:52.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T16:16:52.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T16:16:52.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/5 2024-04-12T16:16:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-12T16:16:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.5 2024-04-12T16:16:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:16:52.073 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/5/whoami 2024-04-12T16:16:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-12T16:16:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T16:16:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T16:16:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T16:16:52.104 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:52.102+0000 7fbdd11fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:16:52.106 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:52.102+0000 7fbdd11fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:16:52.108 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:52.106+0000 7fbdd11fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:16:52.662 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:52.658+0000 7fbdd11fc7c0 -1 Falling back to public interface 2024-04-12T16:16:52.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:52.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:16:53.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:16:54.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:16:54.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:54.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:16:54.496 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:16:54.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:54.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:16:54.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:16:55.770 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:16:55.766+0000 7fbdd11fc7c0 -1 osd.5 83 log_to_monitors true 2024-04-12T16:16:55.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:16:55.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:55.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:16:55.932 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:16:55.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:55.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:16:56.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:16:57.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:16:57.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:16:57.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:16:57.432 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:16:57.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:16:57.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:16:58.060 INFO:tasks.workunit.client.0.smithi137.stderr:osd.5 up in weight 1 up_from 88 up_thru 88 down_at 84 last_clean_interval [30,83) [v2:127.0.0.1:6842/1101001706,v1:127.0.0.1:6843/1101001706] [v2:127.0.0.1:6844/1101001706,v1:127.0.0.1:6845/1101001706] exists,up 5d433deb-975f-44d5-87a0-2f491c33be44 2024-04-12T16:16:58.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:16:58.061 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:16:58.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:16:58.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:16:58.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:16:58.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:16:58.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:16:58.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:16:58.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:16:58.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:16:58.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:16:58.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:58.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:16:59.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-04-12T16:16:59.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-04-12T16:16:59.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-04-12T16:16:59.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:59.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:16:59.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-04-12T16:16:59.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-04-12T16:16:59.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051' 2024-04-12T16:16:59.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:59.276 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:16:59.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509522 2024-04-12T16:16:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509522 2024-04-12T16:16:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522' 2024-04-12T16:16:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:59.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:16:59.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350984 2024-04-12T16:16:59.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350984 2024-04-12T16:16:59.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522 3-347892350984' 2024-04-12T16:16:59.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:59.567 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:16:59.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182467 2024-04-12T16:16:59.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182467 2024-04-12T16:16:59.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522 3-347892350984 4-107374182467' 2024-04-12T16:16:59.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:59.716 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:16:59.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122051 2024-04-12T16:16:59.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122051 2024-04-12T16:16:59.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522 3-347892350984 4-107374182467 5-377957122051' 2024-04-12T16:16:59.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:16:59.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:17:00.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855410 2024-04-12T16:17:00.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855410 2024-04-12T16:17:00.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522 3-347892350984 4-107374182467 5-377957122051 6-150323855410' 2024-04-12T16:17:00.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:00.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:17:00.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691890 2024-04-12T16:17:00.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691890 2024-04-12T16:17:00.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522 3-347892350984 4-107374182467 5-377957122051 6-150323855410 7-171798691890' 2024-04-12T16:17:00.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:00.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:17:00.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528362 2024-04-12T16:17:00.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528362 2024-04-12T16:17:00.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522 3-347892350984 4-107374182467 5-377957122051 6-150323855410 7-171798691890 8-193273528362' 2024-04-12T16:17:00.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:00.296 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:17:00.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364832 2024-04-12T16:17:00.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364832 2024-04-12T16:17:00.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522 3-347892350984 4-107374182467 5-377957122051 6-150323855410 7-171798691890 8-193273528362 9-214748364832' 2024-04-12T16:17:00.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:00.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:17:00.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201301 2024-04-12T16:17:00.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201301 2024-04-12T16:17:00.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509522 3-347892350984 4-107374182467 5-377957122051 6-150323855410 7-171798691890 8-193273528362 9-214748364832 10-236223201301' 2024-04-12T16:17:00.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:00.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-04-12T16:17:00.593 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:00.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:17:00.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-04-12T16:17:00.594 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:00.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-04-12T16:17:00.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-04-12T16:17:00.596 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 21474836574 2024-04-12T16:17:00.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:17:01.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836574 2024-04-12T16:17:01.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:01.006 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-04-12T16:17:01.006 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:01.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:17:01.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-04-12T16:17:01.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-04-12T16:17:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-04-12T16:17:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 42949673051 2024-04-12T16:17:01.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:17:01.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673051 -lt 42949673051 2024-04-12T16:17:01.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:01.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509522 2024-04-12T16:17:01.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:01.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:17:01.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509522 2024-04-12T16:17:01.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:01.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509522 2024-04-12T16:17:01.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-04-12T16:17:01.450 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 64424509522 2024-04-12T16:17:01.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:17:01.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509522 2024-04-12T16:17:01.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:01.889 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-347892350984 2024-04-12T16:17:01.889 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:01.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:17:01.891 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-347892350984 2024-04-12T16:17:01.891 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:01.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350984 2024-04-12T16:17:01.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 347892350984' 2024-04-12T16:17:01.892 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 347892350984 2024-04-12T16:17:01.892 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:17:02.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350984 -lt 347892350984 2024-04-12T16:17:02.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:02.340 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182467 2024-04-12T16:17:02.340 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:02.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:17:02.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182467 2024-04-12T16:17:02.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:02.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182467 2024-04-12T16:17:02.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-04-12T16:17:02.343 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.4 seq 107374182467 2024-04-12T16:17:02.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:17:02.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182468 -lt 107374182467 2024-04-12T16:17:02.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:02.810 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-377957122051 2024-04-12T16:17:02.810 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:02.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:17:02.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-377957122051 2024-04-12T16:17:02.813 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:02.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122051 2024-04-12T16:17:02.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 377957122051' 2024-04-12T16:17:02.814 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.5 seq 377957122051 2024-04-12T16:17:02.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:17:03.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122052 -lt 377957122051 2024-04-12T16:17:03.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:03.303 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855410 2024-04-12T16:17:03.303 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:03.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:17:03.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855410 2024-04-12T16:17:03.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:03.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855410 2024-04-12T16:17:03.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-04-12T16:17:03.306 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.6 seq 150323855410 2024-04-12T16:17:03.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:17:03.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855410 2024-04-12T16:17:03.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:03.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:03.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691890 2024-04-12T16:17:03.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:17:03.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691890 2024-04-12T16:17:03.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:03.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691890 2024-04-12T16:17:03.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-04-12T16:17:03.759 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.7 seq 171798691890 2024-04-12T16:17:03.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:17:04.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691890 2024-04-12T16:17:04.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:04.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528362 2024-04-12T16:17:04.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:04.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:17:04.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528362 2024-04-12T16:17:04.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:04.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528362 2024-04-12T16:17:04.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-04-12T16:17:04.264 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.8 seq 193273528362 2024-04-12T16:17:04.264 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:17:04.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528363 -lt 193273528362 2024-04-12T16:17:04.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:04.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364832 2024-04-12T16:17:04.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:04.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:17:04.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364832 2024-04-12T16:17:04.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:04.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364832 2024-04-12T16:17:04.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364832' 2024-04-12T16:17:04.678 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.9 seq 214748364832 2024-04-12T16:17:04.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:17:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364833 -lt 214748364832 2024-04-12T16:17:05.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:05.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201301 2024-04-12T16:17:05.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:05.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:17:05.146 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201301 2024-04-12T16:17:05.146 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201301 2024-04-12T16:17:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201301' 2024-04-12T16:17:05.147 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.10 seq 236223201301 2024-04-12T16:17:05.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:17:05.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201302 -lt 236223201301 2024-04-12T16:17:05.587 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:17:05.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:17:05.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:17:06.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-12T16:17:06.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:17:06.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:17:06.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:17:06.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:17:06.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:17:06.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:17:06.180 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:17:06.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-12T16:17:06.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:17:06.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:17:06.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:17:07.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-12T16:17:07.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:17:07.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:17:07.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-04-12T16:17:07.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-04-12T16:17:08.125 INFO:tasks.workunit.client.0.smithi137.stdout:crush-device-class= 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:crush-failure-domain=host 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:crush-num-failure-domains=0 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:crush-osds-per-failure-domain=0 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:crush-root=default 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:k=-1 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:l=-1 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:layers=[ [ "cDD", "" ] ] 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:m=-1 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:mapping=_DD 2024-04-12T16:17:08.126 INFO:tasks.workunit.client.0.smithi137.stdout:plugin=lrc 2024-04-12T16:17:08.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-04-12T16:17:08.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-04-12T16:17:09.108 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'remap-pool' already exists 2024-04-12T16:17:09.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:17:10.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-04-12T16:17:10.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-04-12T16:17:10.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2024-04-12T16:17:10.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2024-04-12T16:17:10.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2024-04-12T16:17:10.129 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-04-12T16:17:10.130 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-04-12T16:17:10.151 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-04-12T16:17:10.152 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-04-12T16:17:10.152 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-04-12T16:17:10.153 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-04-12T16:17:10.172 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-04-12T16:17:10.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-04-12T16:17:10.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-04-12T16:17:10.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-04-12T16:17:10.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-04-12T16:17:11.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-04-12T16:17:12.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-04-12T16:17:12.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-04-12T16:17:12.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-04-12T16:17:12.160 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-04-12T16:17:12.160 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:9' 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('4' '6' '9') 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-04-12T16:17:12.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-04-12T16:17:12.810 INFO:tasks.workunit.client.0.smithi137.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-12T16:17:12.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-04-12T16:17:12.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-04-12T16:17:12.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-04-12T16:17:12.938 INFO:tasks.workunit.client.0.smithi137.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-12T16:17:12.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-04-12T16:17:12.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-04-12T16:17:12.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-04-12T16:17:13.066 INFO:tasks.workunit.client.0.smithi137.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-12T16:17:13.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-04-12T16:17:13.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-04-12T16:17:13.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:17:13.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-04-12T16:17:13.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-04-12T16:17:13.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-04-12T16:17:13.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-04-12T16:17:13.074 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-04-12T16:17:13.074 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:17:13.579 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-04-12T16:17:13.579 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:17:13.579 INFO:tasks.workunit.client.0.smithi137.stderr:9' 2024-04-12T16:17:13.579 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('4' '6' '9') 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=6 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=6 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:17:13.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:17:13.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:17:13.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:17:13.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:17:14.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:17:14.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-04-12T16:17:14.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-04-12T16:17:14.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T16:17:14.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=6 2024-04-12T16:17:14.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T16:17:14.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-04-12T16:17:14.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=6 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/6 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-04-12T16:17:15.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:17:15.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:17:15.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/6 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.6 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.6 2024-04-12T16:17:15.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 6 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:17:15.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/6/whoami 2024-04-12T16:17:15.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 6 = 6 ']' 2024-04-12T16:17:15.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T16:17:15.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T16:17:15.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T16:17:15.641 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:15.638+0000 7f644b33d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:17:15.644 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:15.642+0000 7f644b33d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:17:15.645 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:15.642+0000 7f644b33d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 6 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:17:16.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:17:16.188 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:16.186+0000 7f644b33d7c0 -1 Falling back to public interface 2024-04-12T16:17:16.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:17:17.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:17:17.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:17:17.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:17:17.646 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:17:17.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:17:17.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:17:18.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:17:19.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:17:19.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:17:19.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:17:19.118 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:17:19.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:17:19.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:17:19.220 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:19.218+0000 7f644b33d7c0 -1 osd.6 99 log_to_monitors true 2024-04-12T16:17:19.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:17:20.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:17:20.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:17:20.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:17:20.587 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:17:20.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:17:20.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:17:21.379 INFO:tasks.workunit.client.0.smithi137.stderr:osd.6 up in weight 1 up_from 104 up_thru 104 down_at 100 last_clean_interval [35,99) [v2:127.0.0.1:6850/3055370525,v1:127.0.0.1:6851/3055370525] [v2:127.0.0.1:6852/3055370525,v1:127.0.0.1:6853/3055370525] exists,up fc9bb5da-9430-4141-a185-232aec6b8f94 2024-04-12T16:17:21.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:17:21.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:17:21.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:17:21.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T16:17:21.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:17:21.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:17:21.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:17:21.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:17:21.380 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:17:21.380 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:17:21.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:17:21.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:17:21.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:17:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:17:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:17:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:17:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:17:21.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:17:21.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:17:22.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:17:22.693 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:17:22.693 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:17:22.693 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:22.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:17:22.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-04-12T16:17:22.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-04-12T16:17:22.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-04-12T16:17:22.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:22.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:17:23.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673058 2024-04-12T16:17:23.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673058 2024-04-12T16:17:23.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058' 2024-04-12T16:17:23.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:23.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:17:23.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-04-12T16:17:23.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-04-12T16:17:23.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529' 2024-04-12T16:17:23.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:23.243 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:17:23.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350991 2024-04-12T16:17:23.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350991 2024-04-12T16:17:23.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529 3-347892350991' 2024-04-12T16:17:23.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:23.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:17:23.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-04-12T16:17:23.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-04-12T16:17:23.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529 3-347892350991 4-107374182474' 2024-04-12T16:17:23.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:23.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:17:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122058 2024-04-12T16:17:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122058 2024-04-12T16:17:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529 3-347892350991 4-107374182474 5-377957122058' 2024-04-12T16:17:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:23.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:17:23.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-04-12T16:17:23.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-04-12T16:17:23.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529 3-347892350991 4-107374182474 5-377957122058 6-446676598787' 2024-04-12T16:17:23.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:23.853 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:17:24.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691896 2024-04-12T16:17:24.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691896 2024-04-12T16:17:24.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529 3-347892350991 4-107374182474 5-377957122058 6-446676598787 7-171798691896' 2024-04-12T16:17:24.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:24.003 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:17:24.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-04-12T16:17:24.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-04-12T16:17:24.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529 3-347892350991 4-107374182474 5-377957122058 6-446676598787 7-171798691896 8-193273528369' 2024-04-12T16:17:24.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:24.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:17:24.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364839 2024-04-12T16:17:24.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364839 2024-04-12T16:17:24.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529 3-347892350991 4-107374182474 5-377957122058 6-446676598787 7-171798691896 8-193273528369 9-214748364839' 2024-04-12T16:17:24.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:24.291 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:17:24.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201308 2024-04-12T16:17:24.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201308 2024-04-12T16:17:24.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509529 3-347892350991 4-107374182474 5-377957122058 6-446676598787 7-171798691896 8-193273528369 9-214748364839 10-236223201308' 2024-04-12T16:17:24.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:24.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-04-12T16:17:24.440 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:24.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:17:24.441 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-04-12T16:17:24.441 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:24.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-04-12T16:17:24.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-04-12T16:17:24.442 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 21474836580 2024-04-12T16:17:24.443 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:17:24.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836580 2024-04-12T16:17:24.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:24.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:24.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673058 2024-04-12T16:17:24.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:17:24.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673058 2024-04-12T16:17:24.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:24.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673058 2024-04-12T16:17:24.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-04-12T16:17:24.885 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 42949673058 2024-04-12T16:17:24.886 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:17:25.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673058 -lt 42949673058 2024-04-12T16:17:25.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:25.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-04-12T16:17:25.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:25.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:17:25.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-04-12T16:17:25.367 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:25.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-04-12T16:17:25.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-04-12T16:17:25.368 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 64424509529 2024-04-12T16:17:25.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:17:25.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509529 -lt 64424509529 2024-04-12T16:17:25.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:25.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:25.757 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-347892350991 2024-04-12T16:17:25.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:17:25.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-347892350991 2024-04-12T16:17:25.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:25.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350991 2024-04-12T16:17:25.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 347892350991' 2024-04-12T16:17:25.760 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 347892350991 2024-04-12T16:17:25.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:17:26.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350991 -lt 347892350991 2024-04-12T16:17:26.193 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:26.194 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-04-12T16:17:26.194 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:26.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:17:26.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-04-12T16:17:26.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:26.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-04-12T16:17:26.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-04-12T16:17:26.196 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.4 seq 107374182474 2024-04-12T16:17:26.197 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:17:26.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182475 -lt 107374182474 2024-04-12T16:17:26.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:26.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-377957122058 2024-04-12T16:17:26.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:26.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:17:26.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-377957122058 2024-04-12T16:17:26.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:26.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122058 2024-04-12T16:17:26.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 377957122058' 2024-04-12T16:17:26.669 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.5 seq 377957122058 2024-04-12T16:17:26.670 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:17:27.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122059 -lt 377957122058 2024-04-12T16:17:27.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:27.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-446676598787 2024-04-12T16:17:27.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:27.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:17:27.119 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-446676598787 2024-04-12T16:17:27.119 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:27.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-04-12T16:17:27.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 446676598787' 2024-04-12T16:17:27.120 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.6 seq 446676598787 2024-04-12T16:17:27.121 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:17:27.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598787 2024-04-12T16:17:27.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:27.540 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691896 2024-04-12T16:17:27.540 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:27.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:17:27.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691896 2024-04-12T16:17:27.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:27.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691896 2024-04-12T16:17:27.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-04-12T16:17:27.543 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.7 seq 171798691896 2024-04-12T16:17:27.544 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:17:27.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691897 -lt 171798691896 2024-04-12T16:17:27.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:27.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-04-12T16:17:27.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:27.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:17:27.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-04-12T16:17:27.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:27.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-04-12T16:17:27.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-04-12T16:17:27.989 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.8 seq 193273528369 2024-04-12T16:17:27.989 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:17:28.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-04-12T16:17:28.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:28.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364839 2024-04-12T16:17:28.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:28.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:17:28.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364839 2024-04-12T16:17:28.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:28.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364839 2024-04-12T16:17:28.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364839' 2024-04-12T16:17:28.452 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.9 seq 214748364839 2024-04-12T16:17:28.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:17:28.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364839 -lt 214748364839 2024-04-12T16:17:28.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:28.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201308 2024-04-12T16:17:28.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:28.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:17:28.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201308 2024-04-12T16:17:28.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:28.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201308 2024-04-12T16:17:28.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201308' 2024-04-12T16:17:28.887 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.10 seq 236223201308 2024-04-12T16:17:28.887 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:17:29.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201308 -lt 236223201308 2024-04-12T16:17:29.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:17:29.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:17:29.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:17:29.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-04-12T16:17:29.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:17:29.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:17:29.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:17:29.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:17:29.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:17:29.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:17:29.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:17:30.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-04-12T16:17:30.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:17:30.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:17:30.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:17:30.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-04-12T16:17:30.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:17:30.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:17:30.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-04-12T16:17:30.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-04-12T16:17:30.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T16:17:30.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:17:30.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:17:31.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:17:31.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-04-12T16:17:31.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-04-12T16:17:31.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T16:17:31.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-12T16:17:31.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T16:17:31.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-04-12T16:17:31.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-04-12T16:17:32.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-04-12T16:17:32.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/9 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T16:17:32.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:17:32.020 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:17:32.020 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:17:32.020 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:17:32.020 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:17:32.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T16:17:32.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T16:17:32.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/9 2024-04-12T16:17:32.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-12T16:17:32.022 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.9 2024-04-12T16:17:32.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=665acda3-cb3f-42ce-a967-92febb307b06 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48107/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:17:32.022 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/9/whoami 2024-04-12T16:17:32.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-12T16:17:32.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T16:17:32.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T16:17:32.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T16:17:32.054 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:32.050+0000 7fb6ef3027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:17:32.058 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:32.054+0000 7fb6ef3027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:17:32.062 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:32.058+0000 7fb6ef3027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:17:32.614 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:32.610+0000 7fb6ef3027c0 -1 Falling back to public interface 2024-04-12T16:17:32.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-12T16:17:32.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:17:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-12T16:17:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:17:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:17:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:17:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:17:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:17:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:17:32.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:17:33.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:17:34.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:17:34.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:17:34.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:17:34.480 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:17:34.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:17:34.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:17:34.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:17:35.977 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:17:35.974+0000 7fb6ef3027c0 -1 osd.9 107 log_to_monitors true 2024-04-12T16:17:35.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:17:35.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:17:35.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:17:35.982 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:17:35.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:17:35.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:17:36.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:17:37.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:17:37.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:17:37.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:17:37.535 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:17:37.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:17:37.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:osd.9 up in weight 1 up_from 111 up_thru 70 down_at 108 last_clean_interval [50,107) [v2:127.0.0.1:6874/293433558,v1:127.0.0.1:6875/293433558] [v2:127.0.0.1:6876/293433558,v1:127.0.0.1:6877/293433558] exists,up bd763060-03da-4f04-b7aa-a44a886037c6 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:17:38.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:17:38.085 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:17:38.085 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:17:38.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:17:38.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:17:38.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:17:38.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:17:38.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:17:38.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:17:38.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:17:38.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:17:38.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:17:38.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:17:38.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:38.939 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:17:39.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-04-12T16:17:39.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-04-12T16:17:39.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-04-12T16:17:39.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:39.091 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:17:39.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673063 2024-04-12T16:17:39.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673063 2024-04-12T16:17:39.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063' 2024-04-12T16:17:39.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:39.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:17:39.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509534 2024-04-12T16:17:39.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509534 2024-04-12T16:17:39.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534' 2024-04-12T16:17:39.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:39.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:17:39.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350996 2024-04-12T16:17:39.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350996 2024-04-12T16:17:39.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534 3-347892350996' 2024-04-12T16:17:39.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:39.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:17:39.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182479 2024-04-12T16:17:39.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182479 2024-04-12T16:17:39.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534 3-347892350996 4-107374182479' 2024-04-12T16:17:39.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:39.928 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:17:40.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122063 2024-04-12T16:17:40.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122063 2024-04-12T16:17:40.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534 3-347892350996 4-107374182479 5-377957122063' 2024-04-12T16:17:40.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:40.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:17:40.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598793 2024-04-12T16:17:40.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598793 2024-04-12T16:17:40.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534 3-347892350996 4-107374182479 5-377957122063 6-446676598793' 2024-04-12T16:17:40.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:40.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:17:40.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691902 2024-04-12T16:17:40.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691902 2024-04-12T16:17:40.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534 3-347892350996 4-107374182479 5-377957122063 6-446676598793 7-171798691902' 2024-04-12T16:17:40.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:40.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:17:40.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-04-12T16:17:40.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-04-12T16:17:40.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534 3-347892350996 4-107374182479 5-377957122063 6-446676598793 7-171798691902 8-193273528375' 2024-04-12T16:17:40.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:40.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:17:40.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-04-12T16:17:40.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-04-12T16:17:40.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534 3-347892350996 4-107374182479 5-377957122063 6-446676598793 7-171798691902 8-193273528375 9-476741369859' 2024-04-12T16:17:40.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:17:40.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:17:40.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201313 2024-04-12T16:17:40.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201313 2024-04-12T16:17:40.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673063 2-64424509534 3-347892350996 4-107374182479 5-377957122063 6-446676598793 7-171798691902 8-193273528375 9-476741369859 10-236223201313' 2024-04-12T16:17:40.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:40.847 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-04-12T16:17:40.847 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:40.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:17:40.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-04-12T16:17:40.849 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:40.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-04-12T16:17:40.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-04-12T16:17:40.850 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 21474836586 2024-04-12T16:17:40.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:17:41.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836586 2024-04-12T16:17:41.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:41.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673063 2024-04-12T16:17:41.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:41.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:17:41.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673063 2024-04-12T16:17:41.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:41.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673063 2024-04-12T16:17:41.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-04-12T16:17:41.259 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 42949673063 2024-04-12T16:17:41.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:17:41.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673063 2024-04-12T16:17:41.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:41.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509534 2024-04-12T16:17:41.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:41.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:17:41.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509534 2024-04-12T16:17:41.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:41.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509534 2024-04-12T16:17:41.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-04-12T16:17:41.698 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 64424509534 2024-04-12T16:17:41.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:17:42.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509534 2024-04-12T16:17:42.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:42.124 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-347892350996 2024-04-12T16:17:42.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:42.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:17:42.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-347892350996 2024-04-12T16:17:42.126 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:42.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350996 2024-04-12T16:17:42.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 347892350996' 2024-04-12T16:17:42.127 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 347892350996 2024-04-12T16:17:42.128 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:17:42.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350997 -lt 347892350996 2024-04-12T16:17:42.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:42.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182479 2024-04-12T16:17:42.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:42.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:17:42.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182479 2024-04-12T16:17:42.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:42.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182479 2024-04-12T16:17:42.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-04-12T16:17:42.630 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.4 seq 107374182479 2024-04-12T16:17:42.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:17:43.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182480 -lt 107374182479 2024-04-12T16:17:43.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:43.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:43.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-377957122063 2024-04-12T16:17:43.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:17:43.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-377957122063 2024-04-12T16:17:43.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:43.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122063 2024-04-12T16:17:43.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 377957122063' 2024-04-12T16:17:43.056 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.5 seq 377957122063 2024-04-12T16:17:43.056 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:17:43.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122064 -lt 377957122063 2024-04-12T16:17:43.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:43.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-446676598793 2024-04-12T16:17:43.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:43.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:17:43.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-446676598793 2024-04-12T16:17:43.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:43.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598793 2024-04-12T16:17:43.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 446676598793' 2024-04-12T16:17:43.502 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.6 seq 446676598793 2024-04-12T16:17:43.502 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:17:43.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598793 -lt 446676598793 2024-04-12T16:17:43.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:43.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691902 2024-04-12T16:17:43.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:43.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:17:43.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691902 2024-04-12T16:17:43.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:43.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691902 2024-04-12T16:17:43.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691902' 2024-04-12T16:17:43.947 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.7 seq 171798691902 2024-04-12T16:17:43.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:17:44.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691902 -lt 171798691902 2024-04-12T16:17:44.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:44.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-04-12T16:17:44.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:44.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:17:44.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-04-12T16:17:44.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:44.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-04-12T16:17:44.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-04-12T16:17:44.384 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.8 seq 193273528375 2024-04-12T16:17:44.384 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:17:44.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528375 -lt 193273528375 2024-04-12T16:17:44.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:44.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-476741369859 2024-04-12T16:17:44.818 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:44.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:17:44.820 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-476741369859 2024-04-12T16:17:44.820 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:44.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-04-12T16:17:44.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 476741369859' 2024-04-12T16:17:44.822 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.9 seq 476741369859 2024-04-12T16:17:44.822 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:17:45.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369860 -lt 476741369859 2024-04-12T16:17:45.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:17:45.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201313 2024-04-12T16:17:45.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:17:45.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:17:45.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201313 2024-04-12T16:17:45.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:17:45.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201313 2024-04-12T16:17:45.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201313' 2024-04-12T16:17:45.261 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.10 seq 236223201313 2024-04-12T16:17:45.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:17:45.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201314 -lt 236223201313 2024-04-12T16:17:45.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:17:45.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:17:45.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:17:46.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-04-12T16:17:46.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:17:46.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:17:46.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:17:46.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:17:46.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:17:46.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:17:46.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:17:46.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-04-12T16:17:46.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:17:46.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:17:46.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:17:47.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-04-12T16:17:47.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:17:47.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:17:47.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2024-04-12T16:17:47.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=remap-pool 2024-04-12T16:17:47.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-04-12T16:17:47.809 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'remap-pool' does not exist 2024-04-12T16:17:47.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-04-12T16:17:48.707 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile remap-profile does not exist 2024-04-12T16:17:48.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-04-12T16:17:48.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-04-12T16:17:48.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-04-12T16:17:48.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: erasure_code_plugin_exists: local plugin=isa 2024-04-12T16:17:48.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: erasure_code_plugin_exists: local status 2024-04-12T16:17:48.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: erasure_code_plugin_exists: local grepstr 2024-04-12T16:17:48.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: erasure_code_plugin_exists: local s 2024-04-12T16:17:48.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: case `uname` in 2024-04-12T16:17:48.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: uname 2024-04-12T16:17:48.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-04-12T16:17:48.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-04-12T16:17:49.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: s= 2024-04-12T16:17:49.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local status=0 2024-04-12T16:17:49.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-04-12T16:17:49.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-04-12T16:17:50.343 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile TESTPROFILE does not exist 2024-04-12T16:17:50.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: return 0 2024-04-12T16:17:50.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-04-12T16:17:50.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2024-04-12T16:17:50.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-04-12T16:17:50.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-04-12T16:17:50.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-04-12T16:17:52.197 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-isa' already exists 2024-04-12T16:17:52.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:17:53.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-04-12T16:17:53.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:17:53.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-04-12T16:17:53.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-04-12T16:17:54.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:17:54.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:17:54.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-04-12T16:17:54.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-04-12T16:17:54.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-04-12T16:17:54.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:17:54.910 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-04-12T16:17:54.910 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:10 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:9' 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-04-12T16:17:55.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2024-04-12T16:17:55.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-04-12T16:17:55.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-04-12T16:17:55.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-04-12T16:17:56.043 INFO:tasks.workunit.client.0.smithi137.stderr:osd.9 is already out. 2024-04-12T16:17:56.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-04-12T16:18:01.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2024-04-12T16:18:01.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-04-12T16:18:01.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:18:01.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-04-12T16:18:01.065 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-04-12T16:18:01.065 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:10 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:18:01.489 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:18:01.490 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T16:18:01.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-04-12T16:18:01.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:18:01.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:18:01.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-04-12T16:18:02.076 INFO:tasks.workunit.client.0.smithi137.stderr:osd.9 is already in. 2024-04-12T16:18:02.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-04-12T16:18:02.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2024-04-12T16:18:02.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-isa 2024-04-12T16:18:02.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-04-12T16:18:04.013 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-isa' does not exist 2024-04-12T16:18:04.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-04-12T16:18:04.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-04-12T16:18:04.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-04-12T16:18:04.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-04-12T16:18:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-04-12T16:18:04.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:18:05.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:18:05.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-04-12T16:18:05.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-04-12T16:18:05.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-12T16:18:05.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:18:05.312 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-04-12T16:18:05.313 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:18:05.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-04-12T16:18:05.863 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:18:05.863 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:18:05.863 INFO:tasks.workunit.client.0.smithi137.stderr:9' 2024-04-12T16:18:05.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 9 2024-04-12T16:18:05.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2024-04-12T16:18:05.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-04-12T16:18:05.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-04-12T16:18:05.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-04-12T16:18:06.554 INFO:tasks.workunit.client.0.smithi137.stderr:osd.9 is already out. 2024-04-12T16:18:06.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-04-12T16:18:11.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2024-04-12T16:18:11.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-12T16:18:11.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:18:11.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-04-12T16:18:11.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-04-12T16:18:11.575 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:18:12.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-04-12T16:18:12.003 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:18:12.003 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:18:12.003 INFO:tasks.workunit.client.0.smithi137.stderr:5' 2024-04-12T16:18:12.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 5 2024-04-12T16:18:12.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:18:12.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:18:12.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-04-12T16:18:12.604 INFO:tasks.workunit.client.0.smithi137.stderr:osd.9 is already in. 2024-04-12T16:18:12.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-04-12T16:18:12.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-04-12T16:18:12.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-04-12T16:18:12.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-04-12T16:18:14.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-04-12T16:18:14.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-04-12T16:18:15.995 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T16:18:16.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-04-12T16:18:17.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:18:17.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-04-12T16:18:17.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-04-12T16:18:18.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:18:19.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:18:19.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-04-12T16:18:19.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-04-12T16:18:19.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:18:19.414 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:18:19.414 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-04-12T16:18:19.414 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:18:20.139 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-04-12T16:18:20.139 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-04-12T16:18:20.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-04-12T16:18:20.930 INFO:tasks.workunit.client.0.smithi137.stderr:osd.10 is already out. 2024-04-12T16:18:20.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-04-12T16:18:25.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2024-04-12T16:18:25.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:18:25.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:18:25.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2024-04-12T16:18:25.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:18:25.949 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-04-12T16:18:26.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-04-12T16:18:26.368 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:18:26.368 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:18:26.368 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:18:26.368 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:26.368 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:18:26.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 6 2024-04-12T16:18:26.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:18:26.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:18:26.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2024-04-12T16:18:26.962 INFO:tasks.workunit.client.0.smithi137.stderr:osd.10 is already in. 2024-04-12T16:18:26.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-04-12T16:18:26.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-04-12T16:18:26.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2024-04-12T16:18:26.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-04-12T16:18:30.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:18:30.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:18:30.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-04-12T16:18:30.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2024-04-12T16:18:30.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:18:30.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:18:30.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:18:30.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:18:30.650 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:18:30.650 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:18:30.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:18:30.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:18:30.650 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:18:30.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:18:30.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:18:30.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:18:30.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:18:30.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:18:30.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:18:31.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:18:31.243 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:18:31.243 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:18:31.243 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:31.244 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:18:31.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836598 2024-04-12T16:18:31.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836598 2024-04-12T16:18:31.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598' 2024-04-12T16:18:31.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:31.387 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:18:31.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673075 2024-04-12T16:18:31.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673075 2024-04-12T16:18:31.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075' 2024-04-12T16:18:31.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:31.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:18:31.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509546 2024-04-12T16:18:31.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509546 2024-04-12T16:18:31.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546' 2024-04-12T16:18:31.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:31.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:18:31.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892351009 2024-04-12T16:18:31.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892351009 2024-04-12T16:18:31.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546 3-347892351009' 2024-04-12T16:18:31.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:31.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:18:31.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182492 2024-04-12T16:18:31.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182492 2024-04-12T16:18:31.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546 3-347892351009 4-107374182492' 2024-04-12T16:18:31.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:31.985 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:18:32.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122076 2024-04-12T16:18:32.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122076 2024-04-12T16:18:32.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546 3-347892351009 4-107374182492 5-377957122076' 2024-04-12T16:18:32.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:32.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:18:32.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598805 2024-04-12T16:18:32.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598805 2024-04-12T16:18:32.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546 3-347892351009 4-107374182492 5-377957122076 6-446676598805' 2024-04-12T16:18:32.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:32.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:18:32.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691914 2024-04-12T16:18:32.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691914 2024-04-12T16:18:32.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546 3-347892351009 4-107374182492 5-377957122076 6-446676598805 7-171798691914' 2024-04-12T16:18:32.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:32.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:18:32.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528387 2024-04-12T16:18:32.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528387 2024-04-12T16:18:32.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546 3-347892351009 4-107374182492 5-377957122076 6-446676598805 7-171798691914 8-193273528387' 2024-04-12T16:18:32.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:32.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:18:32.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369872 2024-04-12T16:18:32.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369872 2024-04-12T16:18:32.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546 3-347892351009 4-107374182492 5-377957122076 6-446676598805 7-171798691914 8-193273528387 9-476741369872' 2024-04-12T16:18:32.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:32.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:18:32.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201325 2024-04-12T16:18:32.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201325 2024-04-12T16:18:32.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673075 2-64424509546 3-347892351009 4-107374182492 5-377957122076 6-446676598805 7-171798691914 8-193273528387 9-476741369872 10-236223201325' 2024-04-12T16:18:32.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:32.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836598 2024-04-12T16:18:32.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:32.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:18:32.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836598 2024-04-12T16:18:32.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:32.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836598 2024-04-12T16:18:32.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-04-12T16:18:32.938 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836598 2024-04-12T16:18:32.938 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:18:33.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836598 -lt 21474836598 2024-04-12T16:18:33.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:33.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673075 2024-04-12T16:18:33.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:33.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:18:33.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673075 2024-04-12T16:18:33.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:33.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673075 2024-04-12T16:18:33.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2024-04-12T16:18:33.358 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949673075 2024-04-12T16:18:33.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:18:33.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673075 -lt 42949673075 2024-04-12T16:18:33.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:33.778 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509546 2024-04-12T16:18:33.778 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:33.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:18:33.780 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:33.780 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509546 2024-04-12T16:18:33.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509546 2024-04-12T16:18:33.781 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509546 2024-04-12T16:18:33.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509546' 2024-04-12T16:18:33.782 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:18:34.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509547 -lt 64424509546 2024-04-12T16:18:34.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:34.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-347892351009 2024-04-12T16:18:34.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:34.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:18:34.236 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-347892351009 2024-04-12T16:18:34.236 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:34.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892351009 2024-04-12T16:18:34.238 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 347892351009 2024-04-12T16:18:34.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 347892351009' 2024-04-12T16:18:34.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:18:34.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892351009 -lt 347892351009 2024-04-12T16:18:34.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:34.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182492 2024-04-12T16:18:34.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:34.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:18:34.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182492 2024-04-12T16:18:34.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:34.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182492 2024-04-12T16:18:34.701 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 107374182492 2024-04-12T16:18:34.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182492' 2024-04-12T16:18:34.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:18:35.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182492 -lt 107374182492 2024-04-12T16:18:35.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:35.158 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:35.158 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-377957122076 2024-04-12T16:18:35.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:18:35.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-377957122076 2024-04-12T16:18:35.160 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:35.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122076 2024-04-12T16:18:35.162 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 377957122076 2024-04-12T16:18:35.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 377957122076' 2024-04-12T16:18:35.162 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:18:35.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122076 -lt 377957122076 2024-04-12T16:18:35.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:35.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:35.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-446676598805 2024-04-12T16:18:35.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:18:35.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-446676598805 2024-04-12T16:18:35.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:35.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598805 2024-04-12T16:18:35.632 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 446676598805 2024-04-12T16:18:35.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 446676598805' 2024-04-12T16:18:35.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:18:36.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598805 -lt 446676598805 2024-04-12T16:18:36.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:36.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:36.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691914 2024-04-12T16:18:36.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:18:36.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691914 2024-04-12T16:18:36.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:36.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691914 2024-04-12T16:18:36.038 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.7 seq 171798691914 2024-04-12T16:18:36.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691914' 2024-04-12T16:18:36.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:18:36.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691915 -lt 171798691914 2024-04-12T16:18:36.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:36.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528387 2024-04-12T16:18:36.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:36.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:18:36.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528387 2024-04-12T16:18:36.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:36.469 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.8 seq 193273528387 2024-04-12T16:18:36.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528387 2024-04-12T16:18:36.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528387' 2024-04-12T16:18:36.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:18:36.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528388 -lt 193273528387 2024-04-12T16:18:36.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:36.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-476741369872 2024-04-12T16:18:36.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:36.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:18:36.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-476741369872 2024-04-12T16:18:36.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:36.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369872 2024-04-12T16:18:36.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 476741369872' 2024-04-12T16:18:36.911 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.9 seq 476741369872 2024-04-12T16:18:36.911 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:18:37.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369872 -lt 476741369872 2024-04-12T16:18:37.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:37.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201325 2024-04-12T16:18:37.355 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:37.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:18:37.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201325 2024-04-12T16:18:37.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:37.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201325 2024-04-12T16:18:37.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201325' 2024-04-12T16:18:37.359 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.10 seq 236223201325 2024-04-12T16:18:37.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:18:37.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201326 -lt 236223201325 2024-04-12T16:18:37.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:18:37.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:18:37.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:18:38.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-04-12T16:18:38.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:18:38.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:18:38.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:18:38.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:18:38.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:18:38.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:18:38.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:18:38.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-04-12T16:18:38.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:18:38.782 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:18:38.782 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:18:39.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-04-12T16:18:39.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:18:39.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:18:39.299 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-04-12T16:18:39.299 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:18:39.299 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:18:39.299 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-04-12T16:18:39.299 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:18:39.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-04-12T16:18:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:18:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:18:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:18:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:39.704 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:18:39.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-04-12T16:18:39.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-04-12T16:18:39.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2024-04-12T16:18:39.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2024-04-12T16:18:39.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-04-12T16:18:39.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2024-04-12T16:18:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:osd.5 is already out. 2024-04-12T16:18:40.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-04-12T16:18:40.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2024-04-12T16:18:41.744 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 is already out. 2024-04-12T16:18:41.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2024-04-12T16:18:41.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:18:41.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:18:41.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:18:41.765 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:18:41.765 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:18:41.766 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:18:41.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:18:41.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:18:41.766 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:18:41.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:18:41.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:18:41.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:18:41.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:18:41.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:18:41.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:18:42.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:18:42.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:42.753 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:18:43.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836602 2024-04-12T16:18:43.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836602 2024-04-12T16:18:43.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602' 2024-04-12T16:18:43.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:43.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:18:43.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673080 2024-04-12T16:18:43.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673080 2024-04-12T16:18:43.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080' 2024-04-12T16:18:43.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:43.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:18:44.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509551 2024-04-12T16:18:44.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509551 2024-04-12T16:18:44.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551' 2024-04-12T16:18:44.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:44.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:18:44.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892351013 2024-04-12T16:18:44.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892351013 2024-04-12T16:18:44.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551 3-347892351013' 2024-04-12T16:18:44.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:44.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:18:44.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182496 2024-04-12T16:18:44.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182496 2024-04-12T16:18:44.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551 3-347892351013 4-107374182496' 2024-04-12T16:18:44.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:44.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:18:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122080 2024-04-12T16:18:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122080 2024-04-12T16:18:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551 3-347892351013 4-107374182496 5-377957122080' 2024-04-12T16:18:45.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:45.075 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:18:45.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598810 2024-04-12T16:18:45.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598810 2024-04-12T16:18:45.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551 3-347892351013 4-107374182496 5-377957122080 6-446676598810' 2024-04-12T16:18:45.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:45.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:18:45.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691919 2024-04-12T16:18:45.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691919 2024-04-12T16:18:45.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551 3-347892351013 4-107374182496 5-377957122080 6-446676598810 7-171798691919' 2024-04-12T16:18:45.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:45.667 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:18:45.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528392 2024-04-12T16:18:45.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528392 2024-04-12T16:18:45.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551 3-347892351013 4-107374182496 5-377957122080 6-446676598810 7-171798691919 8-193273528392' 2024-04-12T16:18:45.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:45.819 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:18:45.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369876 2024-04-12T16:18:45.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369876 2024-04-12T16:18:45.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551 3-347892351013 4-107374182496 5-377957122080 6-446676598810 7-171798691919 8-193273528392 9-476741369876' 2024-04-12T16:18:45.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:45.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:18:46.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201330 2024-04-12T16:18:46.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201330 2024-04-12T16:18:46.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673080 2-64424509551 3-347892351013 4-107374182496 5-377957122080 6-446676598810 7-171798691919 8-193273528392 9-476741369876 10-236223201330' 2024-04-12T16:18:46.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:46.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836602 2024-04-12T16:18:46.119 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:46.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:18:46.120 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836602 2024-04-12T16:18:46.120 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:46.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836602 2024-04-12T16:18:46.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836602' 2024-04-12T16:18:46.122 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836602 2024-04-12T16:18:46.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:18:46.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836603 -lt 21474836602 2024-04-12T16:18:46.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:46.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673080 2024-04-12T16:18:46.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:46.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:18:46.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673080 2024-04-12T16:18:46.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:46.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673080 2024-04-12T16:18:46.529 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949673080 2024-04-12T16:18:46.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673080' 2024-04-12T16:18:46.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:18:46.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673080 -lt 42949673080 2024-04-12T16:18:46.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:46.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509551 2024-04-12T16:18:46.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:46.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:18:46.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:46.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509551 2024-04-12T16:18:46.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509551 2024-04-12T16:18:46.994 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509551 2024-04-12T16:18:46.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509551' 2024-04-12T16:18:46.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:18:47.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509551 -lt 64424509551 2024-04-12T16:18:47.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:47.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-347892351013 2024-04-12T16:18:47.396 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:47.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:18:47.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-347892351013 2024-04-12T16:18:47.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:47.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892351013 2024-04-12T16:18:47.399 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 347892351013 2024-04-12T16:18:47.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 347892351013' 2024-04-12T16:18:47.400 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:18:47.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892351014 -lt 347892351013 2024-04-12T16:18:47.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:47.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182496 2024-04-12T16:18:47.810 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:47.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:18:47.812 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182496 2024-04-12T16:18:47.812 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:47.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182496 2024-04-12T16:18:47.813 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 107374182496 2024-04-12T16:18:47.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182496' 2024-04-12T16:18:47.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:18:48.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182497 -lt 107374182496 2024-04-12T16:18:48.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:48.210 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-377957122080 2024-04-12T16:18:48.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:48.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:18:48.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-377957122080 2024-04-12T16:18:48.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:48.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122080 2024-04-12T16:18:48.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 377957122080' 2024-04-12T16:18:48.215 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 377957122080 2024-04-12T16:18:48.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:18:48.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122081 -lt 377957122080 2024-04-12T16:18:48.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:48.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-446676598810 2024-04-12T16:18:48.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:48.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:18:48.639 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-446676598810 2024-04-12T16:18:48.639 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:48.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598810 2024-04-12T16:18:48.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 446676598810' 2024-04-12T16:18:48.641 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 446676598810 2024-04-12T16:18:48.641 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:18:49.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598810 -lt 446676598810 2024-04-12T16:18:49.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:49.090 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691919 2024-04-12T16:18:49.091 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:49.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:18:49.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691919 2024-04-12T16:18:49.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:49.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691919 2024-04-12T16:18:49.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691919' 2024-04-12T16:18:49.094 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.7 seq 171798691919 2024-04-12T16:18:49.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:18:49.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691919 -lt 171798691919 2024-04-12T16:18:49.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:49.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528392 2024-04-12T16:18:49.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:49.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:18:49.518 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528392 2024-04-12T16:18:49.518 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:49.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528392 2024-04-12T16:18:49.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528392' 2024-04-12T16:18:49.519 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.8 seq 193273528392 2024-04-12T16:18:49.519 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:18:49.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528392 -lt 193273528392 2024-04-12T16:18:49.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:49.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-476741369876 2024-04-12T16:18:49.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:49.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:18:49.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-476741369876 2024-04-12T16:18:49.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:49.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369876 2024-04-12T16:18:49.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 476741369876' 2024-04-12T16:18:49.991 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.9 seq 476741369876 2024-04-12T16:18:49.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:18:50.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369877 -lt 476741369876 2024-04-12T16:18:50.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:50.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201330 2024-04-12T16:18:50.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:50.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:18:50.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201330 2024-04-12T16:18:50.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:50.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201330 2024-04-12T16:18:50.424 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.10 seq 236223201330 2024-04-12T16:18:50.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201330' 2024-04-12T16:18:50.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:18:50.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201331 -lt 236223201330 2024-04-12T16:18:50.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:18:50.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:18:50.853 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:18:51.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-04-12T16:18:51.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:18:51.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:18:51.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:18:51.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:18:51.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:18:51.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:18:51.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:18:51.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-04-12T16:18:51.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:18:51.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:18:51.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:18:52.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-04-12T16:18:52.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:18:52.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:18:52.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-04-12T16:18:52.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-04-12T16:18:52.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:18:52.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:18:52.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2024-04-12T16:18:52.396 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:18:52.396 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-04-12T16:18:52.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-12T16:18:52.859 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:18:52.859 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:18:52.859 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:18:52.859 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:52.859 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:18:52.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-04-12T16:18:52.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-04-12T16:18:52.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2024-04-12T16:18:52.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-04-12T16:18:52.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:18:52.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:18:52.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-04-12T16:18:52.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:18:53.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-12T16:18:53.327 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:18:53.327 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:18:53.327 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:18:53.327 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:53.327 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:18:53.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-04-12T16:18:53.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:18:53.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:18:53.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-04-12T16:18:53.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2024-04-12T16:18:53.893 INFO:tasks.workunit.client.0.smithi137.stderr:osd.5 is already in. 2024-04-12T16:18:53.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-04-12T16:18:53.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2024-04-12T16:18:55.143 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 is already in. 2024-04-12T16:18:55.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2024-04-12T16:18:55.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:18:55.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:18:55.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:18:55.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:18:55.165 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:18:55.165 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:18:55.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:18:55.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:18:55.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:18:55.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:18:55.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:18:55.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:18:55.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:18:55.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:18:55.269 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:18:56.249 INFO:tasks.workunit.client.0.smithi137.stderr:8 2024-04-12T16:18:56.250 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:18:56.250 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:18:56.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:18:56.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:56.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:18:56.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836607 2024-04-12T16:18:56.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836607 2024-04-12T16:18:56.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607' 2024-04-12T16:18:56.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:56.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:18:57.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673084 2024-04-12T16:18:57.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673084 2024-04-12T16:18:57.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084' 2024-04-12T16:18:57.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:57.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:18:57.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509556 2024-04-12T16:18:57.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509556 2024-04-12T16:18:57.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556' 2024-04-12T16:18:57.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:57.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:18:57.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892351018 2024-04-12T16:18:57.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892351018 2024-04-12T16:18:57.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556 3-347892351018' 2024-04-12T16:18:57.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:57.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:18:58.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182501 2024-04-12T16:18:58.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182501 2024-04-12T16:18:58.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556 3-347892351018 4-107374182501' 2024-04-12T16:18:58.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:58.002 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:18:58.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122085 2024-04-12T16:18:58.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122085 2024-04-12T16:18:58.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556 3-347892351018 4-107374182501 5-377957122085' 2024-04-12T16:18:58.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:58.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:18:58.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598814 2024-04-12T16:18:58.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598814 2024-04-12T16:18:58.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556 3-347892351018 4-107374182501 5-377957122085 6-446676598814' 2024-04-12T16:18:58.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:58.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:18:58.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691923 2024-04-12T16:18:58.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691923 2024-04-12T16:18:58.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556 3-347892351018 4-107374182501 5-377957122085 6-446676598814 7-171798691923' 2024-04-12T16:18:58.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:58.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:18:58.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528396 2024-04-12T16:18:58.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528396 2024-04-12T16:18:58.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556 3-347892351018 4-107374182501 5-377957122085 6-446676598814 7-171798691923 8-193273528396' 2024-04-12T16:18:58.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:58.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-12T16:18:58.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369881 2024-04-12T16:18:58.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369881 2024-04-12T16:18:58.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556 3-347892351018 4-107374182501 5-377957122085 6-446676598814 7-171798691923 8-193273528396 9-476741369881' 2024-04-12T16:18:58.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:18:58.803 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-04-12T16:18:58.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201335 2024-04-12T16:18:58.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201335 2024-04-12T16:18:58.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673084 2-64424509556 3-347892351018 4-107374182501 5-377957122085 6-446676598814 7-171798691923 8-193273528396 9-476741369881 10-236223201335' 2024-04-12T16:18:58.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:58.943 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836607 2024-04-12T16:18:58.943 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:58.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:18:58.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836607 2024-04-12T16:18:58.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:58.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836607 2024-04-12T16:18:58.946 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 21474836607 2024-04-12T16:18:58.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-04-12T16:18:58.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:18:59.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836607 -lt 21474836607 2024-04-12T16:18:59.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:59.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673084 2024-04-12T16:18:59.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:59.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:18:59.415 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673084 2024-04-12T16:18:59.415 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673084 2024-04-12T16:18:59.417 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 42949673084 2024-04-12T16:18:59.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673084' 2024-04-12T16:18:59.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:18:59.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673085 -lt 42949673084 2024-04-12T16:18:59.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:18:59.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509556 2024-04-12T16:18:59.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:18:59.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:18:59.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509556 2024-04-12T16:18:59.825 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:18:59.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509556 2024-04-12T16:18:59.826 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 64424509556 2024-04-12T16:18:59.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509556' 2024-04-12T16:18:59.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:19:00.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509556 -lt 64424509556 2024-04-12T16:19:00.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:19:00.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:19:00.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-347892351018 2024-04-12T16:19:00.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:19:00.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-347892351018 2024-04-12T16:19:00.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:19:00.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892351018 2024-04-12T16:19:00.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 347892351018' 2024-04-12T16:19:00.262 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 347892351018 2024-04-12T16:19:00.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:19:00.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892351018 -lt 347892351018 2024-04-12T16:19:00.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:19:00.705 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:19:00.705 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182501 2024-04-12T16:19:00.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:19:00.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182501 2024-04-12T16:19:00.707 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:19:00.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182501 2024-04-12T16:19:00.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182501' 2024-04-12T16:19:00.708 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 107374182501 2024-04-12T16:19:00.709 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:19:01.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182502 -lt 107374182501 2024-04-12T16:19:01.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:19:01.141 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-377957122085 2024-04-12T16:19:01.141 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:19:01.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:19:01.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-377957122085 2024-04-12T16:19:01.144 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:19:01.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122085 2024-04-12T16:19:01.146 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 377957122085 2024-04-12T16:19:01.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 377957122085' 2024-04-12T16:19:01.146 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:19:01.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122085 -lt 377957122085 2024-04-12T16:19:01.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:19:01.550 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-446676598814 2024-04-12T16:19:01.551 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:19:01.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:19:01.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-446676598814 2024-04-12T16:19:01.554 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:19:01.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598814 2024-04-12T16:19:01.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 446676598814' 2024-04-12T16:19:01.555 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 446676598814 2024-04-12T16:19:01.555 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:19:01.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598815 -lt 446676598814 2024-04-12T16:19:01.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:19:01.984 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691923 2024-04-12T16:19:01.984 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:19:01.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:19:01.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691923 2024-04-12T16:19:01.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:19:01.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691923 2024-04-12T16:19:01.987 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.7 seq 171798691923 2024-04-12T16:19:01.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691923' 2024-04-12T16:19:01.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:19:02.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691924 -lt 171798691923 2024-04-12T16:19:02.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:19:02.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528396 2024-04-12T16:19:02.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:19:02.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:19:02.428 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528396 2024-04-12T16:19:02.428 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:19:02.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528396 2024-04-12T16:19:02.429 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.8 seq 193273528396 2024-04-12T16:19:02.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528396' 2024-04-12T16:19:02.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:19:02.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528397 -lt 193273528396 2024-04-12T16:19:02.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:19:02.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-476741369881 2024-04-12T16:19:02.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:19:02.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-12T16:19:02.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-476741369881 2024-04-12T16:19:02.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:19:02.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369881 2024-04-12T16:19:02.916 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.9 seq 476741369881 2024-04-12T16:19:02.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 476741369881' 2024-04-12T16:19:02.917 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-12T16:19:03.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369882 -lt 476741369881 2024-04-12T16:19:03.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:19:03.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201335 2024-04-12T16:19:03.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:19:03.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-04-12T16:19:03.356 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:19:03.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201335 2024-04-12T16:19:03.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201335 2024-04-12T16:19:03.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201335' 2024-04-12T16:19:03.358 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.10 seq 236223201335 2024-04-12T16:19:03.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-04-12T16:19:03.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201335 -lt 236223201335 2024-04-12T16:19:03.825 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:19:03.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:19:03.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:19:04.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-04-12T16:19:04.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:19:04.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:19:04.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:19:04.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:19:04.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:19:04.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:19:04.399 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:19:04.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-04-12T16:19:04.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:19:04.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:19:04.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:19:05.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-04-12T16:19:05.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:19:05.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:19:05.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-04-12T16:19:05.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:19:05.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:19:05.379 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-04-12T16:19:05.379 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr:10' 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-04-12T16:19:05.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-04-12T16:19:05.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-04-12T16:19:05.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-jerasure 2024-04-12T16:19:05.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T16:19:06.387 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T16:19:06.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-04-12T16:19:07.897 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile profile-jerasure does not exist 2024-04-12T16:19:07.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-04-12T16:19:07.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-04-12T16:19:07.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-04-12T16:19:07.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-04-12T16:19:07.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-04-12T16:19:07.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-04-12T16:19:08.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-04-12T16:19:08.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-04-12T16:19:10.400 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-lrc-a' already exists 2024-04-12T16:19:10.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-04-12T16:19:11.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:11.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-04-12T16:19:11.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:11.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-04-12T16:19:11.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-04-12T16:19:11.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:19:12.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:19:12.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-04-12T16:19:12.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-04-12T16:19:12.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-04-12T16:19:12.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:19:12.184 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-04-12T16:19:12.184 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:19:12.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-04-12T16:19:12.874 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:19:12.874 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T16:19:12.874 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 3 2024-04-12T16:19:12.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2024-04-12T16:19:12.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-04-12T16:19:12.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-04-12T16:19:12.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-04-12T16:19:13.574 INFO:tasks.workunit.client.0.smithi137.stderr:osd.3 is already out. 2024-04-12T16:19:13.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-04-12T16:19:18.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-04-12T16:19:18.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2024-04-12T16:19:18.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-04-12T16:19:18.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:19:18.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-04-12T16:19:18.596 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:19:19.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-04-12T16:19:19.041 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:19:19.041 INFO:tasks.workunit.client.0.smithi137.stderr:7' 2024-04-12T16:19:19.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 7 2024-04-12T16:19:19.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:19:19.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:19:19.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2024-04-12T16:19:19.613 INFO:tasks.workunit.client.0.smithi137.stderr:osd.3 is already in. 2024-04-12T16:19:19.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-04-12T16:19:19.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-04-12T16:19:19.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc-a 2024-04-12T16:19:19.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-04-12T16:19:21.025 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-lrc-a' does not exist 2024-04-12T16:19:21.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-04-12T16:19:22.362 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile profile-lrc-a does not exist 2024-04-12T16:19:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-04-12T16:19:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-04-12T16:19:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-04-12T16:19:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-04-12T16:19:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-04-12T16:19:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-04-12T16:19:23.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-04-12T16:19:23.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-04-12T16:19:24.016 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-lrc' already exists 2024-04-12T16:19:24.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:19:25.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-04-12T16:19:25.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-04-12T16:19:25.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-04-12T16:19:25.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-04-12T16:19:25.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:25.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-04-12T16:19:25.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:25.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-04-12T16:19:25.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:25.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-04-12T16:19:25.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:25.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-04-12T16:19:25.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-04-12T16:19:27.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:19:27.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:19:27.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-04-12T16:19:27.386 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-04-12T16:19:27.386 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-04-12T16:19:27.386 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:19:27.387 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-04-12T16:19:27.387 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:19:27.951 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-04-12T16:19:27.951 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:19:27.951 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:19:27.951 INFO:tasks.workunit.client.0.smithi137.stderr:10 2024-04-12T16:19:27.951 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:19:27.952 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:19:27.952 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:19:27.952 INFO:tasks.workunit.client.0.smithi137.stderr:7' 2024-04-12T16:19:27.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 7 2024-04-12T16:19:27.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2024-04-12T16:19:27.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-04-12T16:19:27.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-04-12T16:19:27.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-04-12T16:19:28.918 INFO:tasks.workunit.client.0.smithi137.stderr:osd.7 is already out. 2024-04-12T16:19:28.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-04-12T16:19:33.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2024-04-12T16:19:33.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2024-04-12T16:19:33.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-04-12T16:19:33.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:19:33.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-04-12T16:19:33.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:19:34.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-04-12T16:19:34.357 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:19:34.357 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:19:34.357 INFO:tasks.workunit.client.0.smithi137.stderr:10 2024-04-12T16:19:34.358 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:19:34.358 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:19:34.358 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:19:34.358 INFO:tasks.workunit.client.0.smithi137.stderr:4' 2024-04-12T16:19:34.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 4 2024-04-12T16:19:34.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:19:34.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:19:34.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2024-04-12T16:19:34.957 INFO:tasks.workunit.client.0.smithi137.stderr:osd.7 is already in. 2024-04-12T16:19:34.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-04-12T16:19:34.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-04-12T16:19:34.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc 2024-04-12T16:19:34.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-04-12T16:19:36.727 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-lrc' does not exist 2024-04-12T16:19:36.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-04-12T16:19:38.151 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile profile-lrc does not exist 2024-04-12T16:19:38.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-04-12T16:19:38.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-04-12T16:19:38.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-04-12T16:19:38.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2024-04-12T16:19:38.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2024-04-12T16:19:38.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-04-12T16:19:39.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-04-12T16:19:39.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-04-12T16:19:40.317 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-shec' already exists 2024-04-12T16:19:40.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:19:41.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-04-12T16:19:41.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-04-12T16:19:41.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-04-12T16:19:41.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-04-12T16:19:41.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-04-12T16:19:42.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:19:43.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:19:43.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-04-12T16:19:43.027 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-04-12T16:19:43.027 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-04-12T16:19:43.028 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:19:43.028 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-04-12T16:19:43.028 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:19:43.467 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-04-12T16:19:43.467 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:19:43.467 INFO:tasks.workunit.client.0.smithi137.stderr:1' 2024-04-12T16:19:43.467 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 1 2024-04-12T16:19:43.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2024-04-12T16:19:43.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-04-12T16:19:43.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-04-12T16:19:43.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-04-12T16:19:44.477 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 is already out. 2024-04-12T16:19:44.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-04-12T16:19:49.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2024-04-12T16:19:49.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-04-12T16:19:49.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-12T16:19:49.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2024-04-12T16:19:49.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-04-12T16:19:49.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:19:49.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-04-12T16:19:49.927 INFO:tasks.workunit.client.0.smithi137.stderr:9 2024-04-12T16:19:49.927 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:19:49.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 6 2024-04-12T16:19:49.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-04-12T16:19:49.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-04-12T16:19:49.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2024-04-12T16:19:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 is already in. 2024-04-12T16:19:50.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-04-12T16:19:50.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2024-04-12T16:19:50.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-shec 2024-04-12T16:19:50.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-04-12T16:19:52.526 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-shec' does not exist 2024-04-12T16:19:52.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-04-12T16:19:54.312 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile profile-shec does not exist 2024-04-12T16:19:54.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-04-12T16:19:54.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=ecpool 2024-04-12T16:19:54.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-04-12T16:19:55.286 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'ecpool' does not exist 2024-04-12T16:19:55.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-04-12T16:19:55.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-04-12T16:19:55.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:19:55.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-04-12T16:19:55.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:19:55.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:19:55.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:19:55.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:19:55.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:19:55.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:19:55.623 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:19:55.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:19:55.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:19:55.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:19:55.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:19:55.627 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:19:55.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:19:55.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:19:55.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:19:55.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:19:55.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:19:55.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:19:55.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-04-12T16:19:55.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:19:55.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:55.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:19:55.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48107 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-code 0 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:19:55.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:19:55.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:19:55.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:19:55.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:19:55.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:19:55.744 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:19:55.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:19:55.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:19:55.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:19:55.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:19:55.746 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:19:55.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:19:55.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:19:55.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:19:55.748 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:19:55.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:19:55.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-12T16:19:55.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-04-12T16:19:55.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:19:55.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:55.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48107 2024-04-12T16:19:55.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48107 2024-04-12T16:19:55.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:19:55.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:19:55.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-04-12T16:19:55.753 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-12T16:19:55.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-12T16:19:55.765 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-04-12T16:19:55.765 DEBUG:teuthology.orchestra.run.smithi137:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf9c5618cc018927fa77780d25b6deb1f5dc254d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-04-12T16:19:55.815 INFO:tasks.workunit.client.0.smithi137.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-12T16:19:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-12T16:19:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-12T16:19:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-12T16:19:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-04-12T16:19:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-04-12T16:19:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-04-12T16:19:55.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-eio 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-04-12T16:19:55.822 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-04-12T16:19:55.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none ' 2024-04-12T16:19:55.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-04-12T16:19:55.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-04-12T16:19:55.823 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-04-12T16:19:55.823 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_ec_object_attr_read_error 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_ec_recovery_multiple_errors 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_ec_recovery_multiple_objects 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_ec_recovery_multiple_objects_eio 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_ec_recovery_unfound 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_ec_single_recovery_error 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_get_bad_size_shard_0 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_get_bad_size_shard_1 2024-04-12T16:19:55.825 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_get_subread_eio_shard_0 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_get_subread_eio_shard_1 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_get_subread_missing 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:19:55.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:19:55.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:19:55.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:19:55.827 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:19:55.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:19:55.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:19:55.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:19:55.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:19:55.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:19:55.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:19:55.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:19:55.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:19:55.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:19:55.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:19:55.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:19:55.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:19:55.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:19:55.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:55.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:19:55.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:19:55.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:19:55.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:19:55.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T16:19:55.835 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:19:55.835 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:55.835 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:19:55.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T16:19:55.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:19:55.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-12T16:19:55.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-12T16:19:55.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:19:55.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T16:19:55.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T16:19:55.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T16:19:55.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:19:55.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:19:55.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:19:55.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T16:19:55.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T16:19:56.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:19:56.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:19:56.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:19:56.474 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:19:56.474 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:56.474 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:19:56.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:19:56.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:19:56.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:19:56.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:19:56.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:19:56.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:19:56.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:19:56.685 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:19:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:19:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:19:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:19:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:19:56.686 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:56.687 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:19:56.687 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:19:56.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:19:56.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T16:19:56.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:19:56.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:19:56.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:19:56.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:19:56.803 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:19:56.803 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:19:56.804 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:19:56.804 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:19:56.804 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:19:56.804 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:56.804 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:19:56.804 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:19:56.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:19:56.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T16:19:56.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T16:19:56.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T16:19:56.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:19:56.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:19:56.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:19:56.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T16:19:56.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:19:57.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:19:57.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:19:57.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:19:57.815 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:19:57.815 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:57.815 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:19:57.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:19:57.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:19:57.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:19:57.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T16:19:57.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:19:58.137 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:19:58.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:19:59.155 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T16:19:59.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:19:59.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:19:59.156 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:19:59.156 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:59.156 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:19:59.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:19:59.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T16:19:59.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T16:19:59.256 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:19:59.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T16:19:59.270 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:19:56.522+0000 7f00591b2c80 0 load: jerasure load: lrc 2024-04-12T16:19:59.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-04-12T16:19:59.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-04-12T16:19:59.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local objname=myobject 2024-04-12T16:19:59.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_backfill_unfound: local lastobj=300 2024-04-12T16:19:59.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:515: TEST_ec_backfill_unfound: local testobj=obj250 2024-04-12T16:19:59.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:19:59.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: setup_osds 5 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:19:59.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:19:59.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:19:59.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9afeab5-8e6a-43f6-96bf-c62f396de458 2024-04-12T16:19:59.282 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 b9afeab5-8e6a-43f6-96bf-c62f396de458 2024-04-12T16:19:59.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b9afeab5-8e6a-43f6-96bf-c62f396de458' 2024-04-12T16:19:59.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:19:59.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvXxlmoa8UEhAA5GfWAd/Twx7F0rffFPBY1A== 2024-04-12T16:19:59.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvXxlmoa8UEhAA5GfWAd/Twx7F0rffFPBY1A=="}' 2024-04-12T16:19:59.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9afeab5-8e6a-43f6-96bf-c62f396de458 -i td/test-erasure-eio/0/new.json 2024-04-12T16:19:59.630 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:19:59.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T16:19:59.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvXxlmoa8UEhAA5GfWAd/Twx7F0rffFPBY1A== --osd-uuid b9afeab5-8e6a-43f6-96bf-c62f396de458 2024-04-12T16:19:59.684 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:19:59.682+0000 7fcd573cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:19:59.707 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:19:59.706+0000 7fcd573cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:19:59.709 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:19:59.706+0000 7fcd573cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:19:59.710 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:19:59.706+0000 7fcd573cb7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:19:59.710 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:19:59.710+0000 7fcd573cb7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:19:59.710 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:19:59.710+0000 7fcd573cb7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:19:59.746 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:19:59.746+0000 7fcd573cb7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T16:20:06.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T16:20:06.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:20:06.384 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:20:06.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:20:06.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:20:06.957 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:20:06.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:20:06.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:20:06.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:20:06.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:20:06.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:20:06.995 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:06.994+0000 7f0955f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:07.004 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:07.002+0000 7f0955f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:07.005 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:07.002+0000 7f0955f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:07.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:20:07.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:20:07.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:20:07.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:20:07.364 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:20:07.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:20:07.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:07.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:20:07.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:07.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:07.575 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:07.574+0000 7f0955f857c0 -1 Falling back to public interface 2024-04-12T16:20:07.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:08.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:08.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:08.768 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:20:08.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:20:08.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:08.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:09.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:10.079 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:10.078+0000 7f0955f857c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:20:10.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:10.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:10.171 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:20:10.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:20:10.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:10.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:10.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:11.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:11.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:11.744 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:20:11.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:20:11.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:11.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:12.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:13.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:13.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:13.194 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:20:13.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:20:13.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:13.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:13.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:14.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:14.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:14.635 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:20:14.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:20:14.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:14.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:15.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:16.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:16.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:16.225 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:20:16.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:20:16.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:16.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:16.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:17.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:17.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:17.656 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:20:17.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:20:17.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:17.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:18.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:19.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:19.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:19.246 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:20:19.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:20:19.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:19.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:19.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:20.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:20.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:20.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:20:20.674 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:20:20.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:20.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:21.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:22.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:22.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:22.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:20:22.138 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:20:22.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:22.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:22.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:23.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:23.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:23.586 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:20:23.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:20:23.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:23.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:24.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:25.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:25.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:25.527 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:20:25.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:20:25.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:25.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:26.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:27.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:27.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:27.376 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:20:27.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:20:27.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:27.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:28.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:29.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:29.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:29.416 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:20:29.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:20:29.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:29.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:30.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:31.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:31.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:31.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:20:31.248 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:20:31.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:31.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:32.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:33.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:33.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:33.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:20:33.181 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:20:33.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:33.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:34.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:35.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:35.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:35.021 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:20:35.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:20:35.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:35.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:20:35.506 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/855723827,v1:127.0.0.1:6803/855723827] [v2:127.0.0.1:6804/855723827,v1:127.0.0.1:6805/855723827] exists,up b9afeab5-8e6a-43f6-96bf-c62f396de458 2024-04-12T16:20:35.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:20:35.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:20:35.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:20:35.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=87b776b2-e1eb-4f06-a9e6-9726c77357a4 2024-04-12T16:20:35.513 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 87b776b2-e1eb-4f06-a9e6-9726c77357a4 2024-04-12T16:20:35.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 87b776b2-e1eb-4f06-a9e6-9726c77357a4' 2024-04-12T16:20:35.513 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:20:35.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTXxlmXg0ZIBAAO6QO+Cey31DW1xIUQsWTig== 2024-04-12T16:20:35.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTXxlmXg0ZIBAAO6QO+Cey31DW1xIUQsWTig=="}' 2024-04-12T16:20:35.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 87b776b2-e1eb-4f06-a9e6-9726c77357a4 -i td/test-erasure-eio/1/new.json 2024-04-12T16:20:36.238 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:20:36.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T16:20:36.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTXxlmXg0ZIBAAO6QO+Cey31DW1xIUQsWTig== --osd-uuid 87b776b2-e1eb-4f06-a9e6-9726c77357a4 2024-04-12T16:20:36.298 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:36.298+0000 7f879cbc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:36.309 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:36.310+0000 7f879cbc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:36.311 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:36.310+0000 7f879cbc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:36.311 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:36.310+0000 7f879cbc77c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:20:36.312 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:36.310+0000 7f879cbc77c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:20:36.312 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:36.310+0000 7f879cbc77c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:20:36.402 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:36.402+0000 7f879cbc77c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T16:20:43.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T16:20:43.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:20:43.157 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:20:43.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:20:43.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:20:43.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:20:43.716 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:20:43.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:20:43.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:20:43.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:20:43.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:20:43.745 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:43.742+0000 7f95caa157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:43.766 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:43.766+0000 7f95caa157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:43.767 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:43.766+0000 7f95caa157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:20:44.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:20:44.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:20:44.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:20:44.127 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:20:44.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:20:44.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:20:44.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:44.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:20:44.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:44.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:44.332 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:44.330+0000 7f95caa157c0 -1 Falling back to public interface 2024-04-12T16:20:44.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:45.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:45.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:45.543 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:20:45.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:20:45.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:45.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:45.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:46.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:46.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:46.959 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:20:46.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:20:46.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:46.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:47.093 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:20:47.090+0000 7f95caa157c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:20:47.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:48.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:48.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:48.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:20:48.402 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:20:48.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:48.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:48.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:49.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:49.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:49.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:20:49.859 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:20:49.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:49.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:50.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:51.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:51.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:51.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:20:51.453 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:20:51.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:51.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:51.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:52.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:52.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:52.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:20:52.882 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:20:52.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:52.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:53.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:54.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:54.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:54.456 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:20:54.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:20:54.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:54.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:54.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:55.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:55.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:55.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:20:55.886 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:20:55.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:55.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:56.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:57.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:57.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:57.428 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:20:57.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:20:57.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:57.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:57.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:20:58.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:20:58.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:20:58.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:20:58.890 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:20:58.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:20:58.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:20:59.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:00.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:00.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:00.432 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:21:00.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:21:00.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:00.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:21:00.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:01.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:01.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:01.875 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:21:01.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:21:01.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:01.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:21:02.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:03.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:03.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:03.702 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:21:03.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:21:03.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:03.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:21:04.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:05.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:05.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:05.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:21:05.570 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:21:05.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:05.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:21:06.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:07.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:07.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:07.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:21:07.477 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:21:07.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:07.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:21:08.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:09.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:09.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:09.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:21:09.586 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:21:09.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:09.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:21:10.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:11.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:11.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:11.403 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:21:11.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:21:11.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:11.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:21:12.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:13.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:13.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:13.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:21:13.423 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:21:13.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:13.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:21:14.028 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1250051528,v1:127.0.0.1:6811/1250051528] [v2:127.0.0.1:6812/1250051528,v1:127.0.0.1:6813/1250051528] exists,up 87b776b2-e1eb-4f06-a9e6-9726c77357a4 2024-04-12T16:21:14.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:21:14.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:21:14.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:21:14.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:21:14.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T16:21:14.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:21:14.032 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:21:14.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:21:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:21:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:21:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:21:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:21:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:21:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:21:14.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:21:14.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:21:14.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=34e44825-6c4a-4e29-8b1b-89c0e685320a 2024-04-12T16:21:14.035 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 34e44825-6c4a-4e29-8b1b-89c0e685320a 2024-04-12T16:21:14.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 34e44825-6c4a-4e29-8b1b-89c0e685320a' 2024-04-12T16:21:14.036 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:21:14.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6Xxlm7dqXAxAAr2yY1RnfJUu+gLGtbouFrg== 2024-04-12T16:21:14.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6Xxlm7dqXAxAAr2yY1RnfJUu+gLGtbouFrg=="}' 2024-04-12T16:21:14.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 34e44825-6c4a-4e29-8b1b-89c0e685320a -i td/test-erasure-eio/2/new.json 2024-04-12T16:21:14.545 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:21:14.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T16:21:14.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6Xxlm7dqXAxAAr2yY1RnfJUu+gLGtbouFrg== --osd-uuid 34e44825-6c4a-4e29-8b1b-89c0e685320a 2024-04-12T16:21:14.601 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:14.602+0000 7fc6c246a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:14.604 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:14.602+0000 7fc6c246a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:14.605 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:14.602+0000 7fc6c246a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:14.605 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:14.606+0000 7fc6c246a7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:21:14.606 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:14.606+0000 7fc6c246a7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:21:14.606 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:14.606+0000 7fc6c246a7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:21:14.702 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:14.702+0000 7fc6c246a7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T16:21:21.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T16:21:21.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:21:21.255 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:21:21.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:21:21.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:21:21.818 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:21:21.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:21:21.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:21:21.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:21:21.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:21:21.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:21:21.854 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:21.854+0000 7fecd64077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:21.856 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:21.854+0000 7fecd64077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:21.857 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:21.858+0000 7fecd64077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:22.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:21:22.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:21:22.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:21:22.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:21:22.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:21:22.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:22.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:21:22.251 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:21:22.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:22.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:22.431 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:22.430+0000 7fecd64077c0 -1 Falling back to public interface 2024-04-12T16:21:22.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:21:23.699 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:21:23.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:23.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:24.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:25.016 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:25.014+0000 7fecd64077c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:21:25.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:25.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:25.119 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:21:25.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:21:25.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:25.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:25.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:26.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:26.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:26.805 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:21:26.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:21:26.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:26.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:27.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:28.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:28.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:28.282 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:21:28.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:21:28.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:28.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:29.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:30.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:30.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:30.011 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:21:30.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:21:30.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:30.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:30.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:31.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:21:31.476 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:21:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:31.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:31.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:32.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:32.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:32.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:21:32.963 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:21:32.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:32.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:33.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:34.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:34.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:34.380 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:21:34.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:21:34.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:34.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:34.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:35.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:35.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:35.930 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:21:35.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:21:35.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:35.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:36.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:37.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:37.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:37.400 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:21:37.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:21:37.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:37.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:37.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:38.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:38.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:38.924 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:21:38.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:21:38.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:38.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:39.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:40.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:40.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:40.840 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:21:40.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:21:40.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:40.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:41.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:42.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:42.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:42.843 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:21:42.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:21:42.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:42.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:43.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:44.929 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:21:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:21:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:45.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:46.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:46.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:46.805 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:21:46.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:21:46.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:46.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:47.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:48.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:48.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:48.258 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:21:48.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:21:48.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:48.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:48.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:21:49.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:21:49.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:49.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:21:49.889 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:21:49.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:49.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:21:50.566 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2726771188,v1:127.0.0.1:6819/2726771188] [v2:127.0.0.1:6820/2726771188,v1:127.0.0.1:6821/2726771188] exists,up 34e44825-6c4a-4e29-8b1b-89c0e685320a 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T16:21:50.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:21:50.570 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:21:50.571 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T16:21:50.572 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:21:50.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d3125570-1d2b-41b5-b110-9447f0cd1c20 2024-04-12T16:21:50.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d3125570-1d2b-41b5-b110-9447f0cd1c20' 2024-04-12T16:21:50.573 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 d3125570-1d2b-41b5-b110-9447f0cd1c20 2024-04-12T16:21:50.573 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:21:50.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeXxlmTLL8IxAAaMMSKzdK0LFqts/K6kPImA== 2024-04-12T16:21:50.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeXxlmTLL8IxAAaMMSKzdK0LFqts/K6kPImA=="}' 2024-04-12T16:21:50.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d3125570-1d2b-41b5-b110-9447f0cd1c20 -i td/test-erasure-eio/3/new.json 2024-04-12T16:21:51.437 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:21:51.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T16:21:51.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCeXxlmTLL8IxAAaMMSKzdK0LFqts/K6kPImA== --osd-uuid d3125570-1d2b-41b5-b110-9447f0cd1c20 2024-04-12T16:21:51.498 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:51.498+0000 7f9b1faf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:51.526 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:51.526+0000 7f9b1faf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:51.528 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:51.526+0000 7f9b1faf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:51.528 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:51.526+0000 7f9b1faf47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:21:51.528 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:51.526+0000 7f9b1faf47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:21:51.529 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:51.526+0000 7f9b1faf47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:21:51.693 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:51.690+0000 7f9b1faf47c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T16:21:58.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T16:21:58.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:21:58.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T16:21:58.441 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T16:21:58.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:21:58.998 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:21:58.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T16:21:58.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:21:58.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:21:58.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:21:58.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:21:59.035 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:59.034+0000 7fb8ab3c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:59.058 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:59.058+0000 7fb8ab3c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:59.059 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:59.058+0000 7fb8ab3c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:21:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T16:21:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:21:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:21:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:21:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:21:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:21:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:21:59.443 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:21:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:21:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:21:59.631 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:21:59.630+0000 7fb8ab3c07c0 -1 Falling back to public interface 2024-04-12T16:21:59.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:00.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:00.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:00.900 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:22:00.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:22:00.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:00.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:01.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:02.314 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:02.314+0000 7fb8ab3c07c0 -1 osd.3 0 log_to_monitors true 2024-04-12T16:22:02.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:02.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:02.340 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:22:02.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:22:02.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:02.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:03.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:04.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:04.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:04.182 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:22:04.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:22:04.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:04.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:04.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:05.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:05.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:05.997 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:22:05.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:22:05.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:05.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:06.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:07.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:07.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:07.775 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:22:07.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:22:07.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:07.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:08.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:09.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:09.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:09.285 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:22:09.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:22:09.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:09.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:09.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:10.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:10.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:10.826 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:22:10.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:22:10.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:10.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:11.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:12.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:12.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:12.416 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:22:12.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:22:12.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:12.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:12.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:13.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:13.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:13.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:22:13.913 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:22:13.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:13.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:14.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:15.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:15.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:15.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:22:15.436 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:22:15.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:15.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:15.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:16.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:16.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:16.863 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:22:16.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:22:16.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:16.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:18.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:19.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:19.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:19.046 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:22:19.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:22:19.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:19.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:19.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:20.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:22:20.642 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:22:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:20.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:21.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:22.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:22.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:22.823 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:22:22.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:22:22.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:22.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:24.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:25.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:25.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:25.094 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:22:25.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:22:25.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:25.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:25.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:26.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:26.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:26.630 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:22:26.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:22:26.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:26.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:27.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:28.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:28.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:28.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:22:28.478 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:22:28.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:28.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:22:28.900 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2609729727,v1:127.0.0.1:6827/2609729727] [v2:127.0.0.1:6828/2609729727,v1:127.0.0.1:6829/2609729727] exists,up d3125570-1d2b-41b5-b110-9447f0cd1c20 2024-04-12T16:22:28.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:22:28.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:22:28.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:22:28.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:22:28.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-04-12T16:22:28.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:22:28.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff193eea-d591-4d22-8bc9-bc8e9c54763e 2024-04-12T16:22:28.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 ff193eea-d591-4d22-8bc9-bc8e9c54763e' 2024-04-12T16:22:28.907 INFO:tasks.workunit.client.0.smithi137.stdout:add osd4 ff193eea-d591-4d22-8bc9-bc8e9c54763e 2024-04-12T16:22:28.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:22:28.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDEXxlmnXOENxAActFa7BotKswSqhgS84LjwQ== 2024-04-12T16:22:28.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDEXxlmnXOENxAActFa7BotKswSqhgS84LjwQ=="}' 2024-04-12T16:22:28.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff193eea-d591-4d22-8bc9-bc8e9c54763e -i td/test-erasure-eio/4/new.json 2024-04-12T16:22:29.796 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:22:29.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-04-12T16:22:29.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEXxlmnXOENxAActFa7BotKswSqhgS84LjwQ== --osd-uuid ff193eea-d591-4d22-8bc9-bc8e9c54763e 2024-04-12T16:22:29.858 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:29.858+0000 7fe0498af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:22:29.918 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:29.918+0000 7fe0498af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:22:29.919 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:29.918+0000 7fe0498af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:22:29.920 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:29.918+0000 7fe0498af7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:22:29.920 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:29.918+0000 7fe0498af7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:22:29.920 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:29.918+0000 7fe0498af7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:22:30.135 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:30.134+0000 7fe0498af7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-04-12T16:22:37.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-04-12T16:22:37.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:22:37.347 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd4 key to auth repository 2024-04-12T16:22:37.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-12T16:22:37.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:22:37.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-12T16:22:37.920 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.4 2024-04-12T16:22:37.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:22:37.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:22:37.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:22:37.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:22:37.957 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:37.958+0000 7f79dadb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:22:38.022 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:38.022+0000 7f79dadb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:22:38.024 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:38.022+0000 7f79dadb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:22:38.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-12T16:22:38.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:22:38.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-12T16:22:38.339 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:22:38.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:22:38.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:22:38.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:38.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:22:38.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:38.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:38.595 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:38.594+0000 7f79dadb87c0 -1 Falling back to public interface 2024-04-12T16:22:38.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:39.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:39.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:39.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:22:39.779 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:22:39.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:39.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:40.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:41.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:41.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:41.189 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:22:41.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:22:41.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:41.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:41.252 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:22:41.250+0000 7f79dadb87c0 -1 osd.4 0 log_to_monitors true 2024-04-12T16:22:41.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:42.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:42.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:42.820 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:22:42.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:22:42.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:42.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:43.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:44.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:44.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:44.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:22:44.537 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:22:44.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:44.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:45.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:46.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:46.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:46.229 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:22:46.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:22:46.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:46.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:46.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:47.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:47.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:47.742 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:22:47.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:22:47.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:47.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:48.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:49.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:49.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:49.275 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:22:49.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:22:49.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:49.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:49.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:50.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:50.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:50.713 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:22:50.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:22:50.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:50.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:51.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:52.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:52.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:52.271 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:22:52.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:22:52.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:52.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:52.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:53.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:53.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:53.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:22:53.713 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:22:53.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:53.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:54.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:55.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:55.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:55.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:22:55.184 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:22:55.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:55.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:55.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:56.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:22:56.626 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:22:56.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:56.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:57.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:22:58.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:22:58.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:22:58.712 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:22:58.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:22:58.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:22:58.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:22:59.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:23:00.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:23:00.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:23:00.269 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:23:00.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:23:00.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:23:00.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:23:00.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:23:01.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:23:01.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:23:01.916 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:23:01.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:23:01.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:23:01.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:23:02.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:23:03.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:23:03.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:23:03.716 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:23:03.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:23:03.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:23:03.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:23:04.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:23:05.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:23:05.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:23:05.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:23:05.489 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:23:05.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:23:05.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:23:06.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:23:07.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:23:07.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:23:07.189 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:23:07.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:23:07.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:23:07.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:23:07.637 INFO:tasks.workunit.client.0.smithi137.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2513758389,v1:127.0.0.1:6835/2513758389] [v2:127.0.0.1:6836/2513758389,v1:127.0.0.1:6837/2513758389] exists,up ff193eea-d591-4d22-8bc9-bc8e9c54763e 2024-04-12T16:23:07.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:23:07.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:23:07.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:23:07.638 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T16:23:07.638 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:23:07.638 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:23:07.639 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:23:07.639 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:23:07.639 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:23:07.639 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T16:23:07.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T16:23:07.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T16:23:07.738 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:23:07.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T16:23:07.757 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:20:07.578+0000 7f0955f857c0 0 load: jerasure load: lrc 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T16:23:07.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-04-12T16:23:09.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T16:23:09.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T16:23:10.572 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T16:23:10.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:23:11.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T16:23:11.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:23:11.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:23:11.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:23:11.597 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:23:11.597 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:23:11.598 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:23:11.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:23:11.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:23:11.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:23:11.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:23:11.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:23:11.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:23:11.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:23:11.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:23:11.721 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:23:12.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:23:12.152 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:23:12.153 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:23:12.153 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:23:12.153 INFO:tasks.workunit.client.0.smithi137.stderr:4' 2024-04-12T16:23:12.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:23:12.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:23:12.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:23:12.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803813 2024-04-12T16:23:12.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803813 2024-04-12T16:23:12.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813' 2024-04-12T16:23:12.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:23:12.349 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:23:12.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574876 2024-04-12T16:23:12.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574876 2024-04-12T16:23:12.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574876' 2024-04-12T16:23:12.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:23:12.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:23:12.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378649 2024-04-12T16:23:12.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378649 2024-04-12T16:23:12.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574876 2-81604378649' 2024-04-12T16:23:12.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:23:12.712 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:23:12.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084307 2024-04-12T16:23:12.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084307 2024-04-12T16:23:12.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574876 2-81604378649 3-120259084307' 2024-04-12T16:23:12.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:23:12.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:23:13.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757258 2024-04-12T16:23:13.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757258 2024-04-12T16:23:13.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574876 2-81604378649 3-120259084307 4-163208757258' 2024-04-12T16:23:13.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:23:13.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803813 2024-04-12T16:23:13.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:23:13.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:23:13.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803813 2024-04-12T16:23:13.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:23:13.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803813 2024-04-12T16:23:13.016 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803813 2024-04-12T16:23:13.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-04-12T16:23:13.016 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:23:13.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803813 -lt 25769803813 2024-04-12T16:23:13.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:23:13.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574876 2024-04-12T16:23:13.417 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:23:13.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:23:13.418 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574876 2024-04-12T16:23:13.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:23:13.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574876 2024-04-12T16:23:13.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2024-04-12T16:23:13.420 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574876 2024-04-12T16:23:13.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:23:13.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574874 -lt 55834574876 2024-04-12T16:23:13.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T16:23:14.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T16:23:14.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:23:15.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574877 -lt 55834574876 2024-04-12T16:23:15.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:23:15.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378649 2024-04-12T16:23:15.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:23:15.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:23:15.388 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378649 2024-04-12T16:23:15.388 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:23:15.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378649 2024-04-12T16:23:15.389 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378649 2024-04-12T16:23:15.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378649' 2024-04-12T16:23:15.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:23:15.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378649 -lt 81604378649 2024-04-12T16:23:15.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:23:15.825 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084307 2024-04-12T16:23:15.825 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:23:15.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:23:15.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084307 2024-04-12T16:23:15.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:23:15.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084307 2024-04-12T16:23:15.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084307' 2024-04-12T16:23:15.829 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 120259084307 2024-04-12T16:23:15.830 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:23:16.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084308 -lt 120259084307 2024-04-12T16:23:16.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:23:16.258 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757258 2024-04-12T16:23:16.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:23:16.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:23:16.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:23:16.262 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757258 2024-04-12T16:23:16.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757258 2024-04-12T16:23:16.263 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 163208757258 2024-04-12T16:23:16.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757258' 2024-04-12T16:23:16.264 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:23:16.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757258 -lt 163208757258 2024-04-12T16:23:16.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:23:16.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:23:16.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:23:17.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:23:17.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:23:17.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:23:17.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:23:17.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:23:17.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:23:17.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:23:17.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:23:17.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:23:17.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:23:17.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:23:17.692 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:23:18.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:23:18.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:23:18.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:23:18.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:525: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:12.323433+0000 0'0 45:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-04-12T16:23:09.955422+0000 0'0 2024-04-12T16:23:09.955422+0000 0 0 periodic scrub scheduled @ 2024-04-14T03:44:48.319807+0000 0 0 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.909074+0000 0'0 46:19 [4,1,2] 4 [4,1,2] 4 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-13T18:44:04.998607+0000 0 0 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.909110+0000 0'0 45:102 [0,4,1] 0 [0,4,1] 0 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T00:07:12.671207+0000 0 0 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.398019+0000 0'0 46:19 [4,3,0] 4 [4,3,0] 4 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T03:37:48.051461+0000 0 0 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:21:51.034316+0000 0'0 45:74 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-13T22:55:11.767705+0000 0 0 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-12T16:23:18.635 INFO:tasks.workunit.client.0.smithi137.stderr:dumped pgs 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:527: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:23:18.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:23:18.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:23:18.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:23:18.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:18.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-04-12T16:23:18.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-04-12T16:23:18.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-04-12T16:23:18.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:23:18.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-12T16:23:19.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local primary=3 2024-04-12T16:23:19.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-04-12T16:23:19.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:23:19.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T16:23:19.265 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:23:19.266 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:23:19.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:23:19.919 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:23:19.919 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:23:19.919 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:23:19.919 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T16:23:19.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-04-12T16:23:19.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-04-12T16:23:19.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: local -a initial_osds 2024-04-12T16:23:19.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local last_osd=2 2024-04-12T16:23:19.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-04-12T16:23:19.921 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:23:19.921 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:23:19.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:23:19.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:23:19.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:23:20.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:23:20.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: ceph osd down 2 2024-04-12T16:23:21.204 INFO:tasks.workunit.client.0.smithi137.stderr:osd.2 is already down. 2024-04-12T16:23:21.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph osd out 2 2024-04-12T16:23:21.847 INFO:tasks.workunit.client.0.smithi137.stderr:osd.2 is already out. 2024-04-12T16:23:21.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-04-12T16:23:22.433 INFO:tasks.workunit.client.0.smithi137.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-12T16:23:22.433 INFO:tasks.workunit.client.0.smithi137.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 active+clean 2024-04-12T16:23:12.323433+0000 46'1 46:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-04-12T16:23:09.955422+0000 0'0 2024-04-12T16:23:09.955422+0000 0 0 periodic scrub scheduled @ 2024-04-14T02:26:36.437207+0000 0 0 2024-04-12T16:23:22.434 INFO:tasks.workunit.client.0.smithi137.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.909074+0000 0'0 46:19 [4,1,2] 4 [4,1,2] 4 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-13T18:44:04.998607+0000 0 0 2024-04-12T16:23:22.434 INFO:tasks.workunit.client.0.smithi137.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.909110+0000 0'0 47:106 [0,4,1] 0 [0,4,1] 0 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T00:07:12.671207+0000 0 0 2024-04-12T16:23:22.434 INFO:tasks.workunit.client.0.smithi137.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.398019+0000 0'0 46:19 [4,3,0] 4 [4,3,0] 4 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T03:37:48.051461+0000 0 0 2024-04-12T16:23:22.434 INFO:tasks.workunit.client.0.smithi137.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:21:51.034316+0000 0'0 46:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-13T22:55:11.767705+0000 0 0 2024-04-12T16:23:22.434 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-04-12T16:23:22.434 INFO:tasks.workunit.client.0.smithi137.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-12T16:23:22.434 INFO:tasks.workunit.client.0.smithi137.stderr:dumped pgs 2024-04-12T16:23:22.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-04-12T16:23:22.452 INFO:tasks.workunit.client.0.smithi137.stderr:4+0 records in 2024-04-12T16:23:22.452 INFO:tasks.workunit.client.0.smithi137.stderr:4+0 records out 2024-04-12T16:23:22.452 INFO:tasks.workunit.client.0.smithi137.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000301551 s, 13.6 MB/s 2024-04-12T16:23:22.453 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: seq 1 300 2024-04-12T16:23:22.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:22.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:22.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:22.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:23.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:23.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:23.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:23.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:23.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:23.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:24.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:24.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:24.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:24.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:24.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:24.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:25.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:25.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:25.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:25.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:25.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:25.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:26.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:26.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:26.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:26.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:26.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:26.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:27.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:27.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:27.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:27.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:27.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:27.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:27.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:27.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:28.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:28.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:28.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:28.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:28.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:28.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:28.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:28.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:29.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:29.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:29.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:29.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:29.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:29.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:29.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:29.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:30.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:30.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:30.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:30.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:30.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:31.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:31.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:31.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:31.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:31.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:31.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:31.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:31.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:32.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:32.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:32.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:32.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:32.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:32.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:33.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:33.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:33.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:33.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:33.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:33.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:33.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:33.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:34.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:34.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:34.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:34.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:34.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:34.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:35.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:35.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:35.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:35.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:35.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:35.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:35.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:35.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:36.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:36.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:36.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:36.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:36.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:36.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:36.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:36.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:37.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:37.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:37.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:37.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:37.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:37.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:38.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:38.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:38.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:38.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:38.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:38.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:38.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:38.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:39.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:39.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:39.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:39.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:39.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:39.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:40.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:40.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:40.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:40.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:40.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:40.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:40.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:40.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:41.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:41.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:41.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:41.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:41.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:41.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:41.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:41.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:42.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:42.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:42.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:42.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:42.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:42.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:43.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:43.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:43.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:43.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:43.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:43.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:43.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:43.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:44.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:44.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:44.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:44.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:44.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:44.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:45.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:45.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:45.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:45.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:45.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:45.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:45.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:45.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:46.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:46.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:46.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:46.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:46.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:46.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:47.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:47.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:47.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:47.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:47.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:47.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:47.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:47.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:48.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:48.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:48.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:48.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:48.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:48.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:48.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:48.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:49.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:49.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:49.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:49.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:49.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:49.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:50.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:50.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:50.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:50.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:50.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:50.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:50.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:50.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:51.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:51.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:51.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:51.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:51.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:51.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:52.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:52.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:52.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:52.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:52.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:52.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:53.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:53.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:53.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:53.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:53.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:53.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:54.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:54.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:54.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:54.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:54.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:54.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:54.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:54.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:55.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:55.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:55.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:55.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:55.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:55.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:56.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:56.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:56.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:56.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:56.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:56.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:56.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:56.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:57.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:57.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:57.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:57.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:57.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:57.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:58.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:58.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:58.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:58.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:58.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:58.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:58.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:58.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:59.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:59.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:59.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:59.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:59.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:59.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-04-12T16:23:59.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:23:59.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:00.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:00.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:00.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:00.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:00.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:00.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:01.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:01.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:01.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:01.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:01.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:01.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:02.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:02.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:02.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:02.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:02.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:02.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:03.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:03.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:03.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:03.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:03.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:03.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:04.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:04.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:04.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:04.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:04.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:04.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:05.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:05.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:05.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:05.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:05.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:05.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:05.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:05.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:06.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:06.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:06.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:06.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:06.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:06.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:06.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:06.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:07.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:07.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:07.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:07.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:07.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:07.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:08.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:08.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:08.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:08.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:08.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:08.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:08.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:08.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:09.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:09.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:09.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:09.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:09.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:09.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:10.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:10.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:10.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:10.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:10.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:10.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:11.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:11.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:11.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:11.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:11.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:11.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:11.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:11.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:12.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:12.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:12.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:12.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:12.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:12.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:12.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:13.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:13.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:13.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:13.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:13.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:13.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:14.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:14.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:14.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:14.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:14.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:14.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:14.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:14.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:15.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:15.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:15.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:15.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:15.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:15.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:15.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:15.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:16.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:16.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:16.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:16.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:16.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:16.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:17.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:17.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:17.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:17.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:17.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:17.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:17.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:17.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:18.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:18.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:18.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:18.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:18.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:18.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:19.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:19.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:19.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:19.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:19.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:19.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:19.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:19.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:20.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:20.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:20.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:20.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:20.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:20.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:21.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:21.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:21.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:21.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:21.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:21.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:21.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:21.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:22.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:22.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:22.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:22.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:22.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:22.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:22.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:22.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:23.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:23.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:23.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:23.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:23.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:23.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:24.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:24.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:24.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:24.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:24.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:24.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:24.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:24.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:25.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:25.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:25.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:25.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:25.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:25.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:25.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:25.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:26.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:26.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:26.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:26.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:26.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:26.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:27.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:27.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:27.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:27.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:27.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:27.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:27.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:27.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:28.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:28.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:28.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:28.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:28.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:28.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:29.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:29.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:29.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:29.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:29.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:29.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:29.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:29.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:30.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:30.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:30.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:30.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:30.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:30.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:31.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:31.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:31.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:31.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:31.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:31.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:31.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:31.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:32.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:32.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:32.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:32.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:32.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:32.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:33.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:33.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:33.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:33.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:33.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:33.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:34.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:34.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:34.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:34.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:34.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:34.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:34.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:34.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:35.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:35.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:35.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:35.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:35.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:35.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:36.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:36.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:36.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:36.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:36.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:36.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:36.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:36.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:37.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:37.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:37.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:37.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:37.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:37.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:37.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:37.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:38.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:38.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:38.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:38.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:38.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:38.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:39.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:39.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:39.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:39.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:39.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:39.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:40.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:40.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:40.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:40.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:40.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:40.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:40.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:40.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:41.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:41.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:41.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:41.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:41.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:41.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:41.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:41.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:42.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:42.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:42.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:42.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:42.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:42.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:43.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:43.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:43.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:43.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:43.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:43.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:43.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:43.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:44.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:44.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:44.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:44.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:44.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:44.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:45.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:45.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:45.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:45.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:45.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:45.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:46.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:46.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:46.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:46.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:46.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:46.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:46.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:24:46.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-04-12T16:24:47.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:544: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj250 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:24:47.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj250 2024-04-12T16:24:47.072 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:24:47.072 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-12T16:24:47.072 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-04-12T16:24:47.072 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:24:47.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr:2147483647' 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:24:47.550 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-04-12T16:24:47.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:24:47.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-12T16:24:47.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:24:47.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-12T16:24:47.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:24:47.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:24:47.552 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:24:47.552 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-12T16:24:47.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T16:24:47.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T16:24:47.553 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:24:47.553 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:24:47.553 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:24:47.553 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T16:24:47.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:24:47.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:24:47.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:24:47.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:24:47.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-04-12T16:24:47.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T16:24:47.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T16:24:47.679 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:24:47.679 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:24:47.679 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:24:47.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T16:24:47.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:24:47.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj250 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:24:47.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-04-12T16:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj250 2024-04-12T16:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-12T16:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:24:47.800 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr:2147483647' 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:24:48.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-04-12T16:24:48.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:24:48.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-04-12T16:24:48.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:24:48.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-12T16:24:48.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:24:48.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-12T16:24:48.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:24:48.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:24:48.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:24:48.202 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-12T16:24:48.202 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-12T16:24:48.202 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-12T16:24:48.203 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:24:48.203 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:24:48.203 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:24:48.203 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.1.asok 2024-04-12T16:24:48.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:24:48.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:24:48.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:24:48.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:24:48.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-04-12T16:24:48.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-12T16:24:48.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-12T16:24:48.313 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:24:48.313 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:24:48.313 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:24:48.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.1.asok 2024-04-12T16:24:48.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:24:48.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:24:48.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:24:48.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:24:48.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:24:48.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:24:48.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:24:48.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:24:48.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:24:48.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:24:48.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:24:48.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:24:48.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T16:24:48.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T16:24:48.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:24:48.442 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:24:48.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-12T16:24:48.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:24:48.443 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-04-12T16:24:48.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-12T16:24:48.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T16:24:48.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T16:24:48.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T16:24:48.477 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:24:48.477+0000 7fc4b77b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:24:48.486 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:24:48.485+0000 7fc4b77b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:24:48.488 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:24:48.485+0000 7fc4b77b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:24:48.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-12T16:24:48.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:24:48.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:24:48.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:24:48.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:24:48.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:24:48.852 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:24:48.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:24:48.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:24:48.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:24:49.039 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:24:49.037+0000 7fc4b77b07c0 -1 Falling back to public interface 2024-04-12T16:24:49.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:24:50.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:24:50.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:24:50.265 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:24:50.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:24:50.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:24:50.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:24:50.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:24:51.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:24:51.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:24:51.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:24:51.707 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:24:51.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:24:51.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:24:52.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:24:52.134 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:24:52.133+0000 7fc4b77b07c0 -1 osd.2 46 log_to_monitors true 2024-04-12T16:24:53.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:24:53.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:24:53.114 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:24:53.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:24:53.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:24:53.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:24:53.579 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up out weight 0 up_from 54 up_thru 19 down_at 47 last_clean_interval [19,46) [v2:127.0.0.1:6818/537801974,v1:127.0.0.1:6819/537801974] [v2:127.0.0.1:6820/537801974,v1:127.0.0.1:6821/537801974] exists,up 34e44825-6c4a-4e29-8b1b-89c0e685320a 2024-04-12T16:24:53.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:24:53.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:24:53.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:24:53.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: ceph osd in 2 2024-04-12T16:24:54.652 INFO:tasks.workunit.client.0.smithi137.stderr:osd.2 is already in. 2024-04-12T16:24:54.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: sleep 15 2024-04-12T16:25:09.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: seq 1 240 2024-04-12T16:25:09.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:09.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:09.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:09.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:09.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:09.679 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:10.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:10.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:10.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:10.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:10.284 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:10.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:10.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:11.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:11.286 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:11.286 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:11.286 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:11.286 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:11.286 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:11.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:11.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:11.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:11.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:11.716 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:11.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:11.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:12.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:12.718 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:12.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:12.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:12.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:12.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:13.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:13.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:13.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:13.132 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:13.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:13.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:13.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:14.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:14.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:14.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:14.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:14.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:14.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:14.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:14.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:14.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:14.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:14.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:14.568 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:14.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:15.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:15.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:15.571 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:15.571 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:15.571 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:15.571 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:15.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:15.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:15.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:15.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:15.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:15.999 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:15.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:17.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:17.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:17.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:17.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:17.002 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:17.002 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:17.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:17.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:17.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:17.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:17.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:17.419 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:17.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:18.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:18.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:18.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:18.421 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:18.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:18.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:18.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:18.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:18.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:18.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:18.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:18.854 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:18.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:19.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:19.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:19.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:19.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:19.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:19.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:20.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:20.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:20.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:20.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:20.288 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:20.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:20.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:21.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:21.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:21.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:21.289 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:21.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:21.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:21.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:21.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:21.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:21.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:21.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:21.712 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:21.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:22.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:22.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:22.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:22.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:22.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:22.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:23.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:23.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:23.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:23.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-04-12T16:25:23.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:23.133 INFO:tasks.workunit.client.0.smithi137.stdout:active+undersized+degraded+remapped+backfilling 2024-04-12T16:25:23.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-04-12T16:25:24.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-04-12T16:25:24.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-04-12T16:25:24.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:25:24.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:25:24.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:25:24.136 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:25:24.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-04-12T16:25:24.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-04-12T16:25:24.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-04-12T16:25:24.545 INFO:tasks.workunit.client.0.smithi137.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-04-12T16:25:24.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-04-12T16:25:24.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: break 2024-04-12T16:25:24.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:562: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-04-12T16:25:24.948 INFO:tasks.workunit.client.0.smithi137.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-12T16:25:24.949 INFO:tasks.workunit.client.0.smithi137.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-04-12T16:25:22.094072+0000 52'301 58:900 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-04-12T16:23:09.955422+0000 0'0 2024-04-12T16:23:09.955422+0000 0 0 no scrub is scheduled 0 0 2024-04-12T16:25:24.949 INFO:tasks.workunit.client.0.smithi137.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:24:55.482882+0000 0'0 58:56 [4,1,2] 4 [4,1,2] 4 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-13T16:43:37.398941+0000 0 0 2024-04-12T16:25:24.949 INFO:tasks.workunit.client.0.smithi137.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.909110+0000 0'0 58:127 [0,4,1] 0 [0,4,1] 0 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T00:07:12.671207+0000 0 0 2024-04-12T16:25:24.949 INFO:tasks.workunit.client.0.smithi137.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.398019+0000 0'0 58:43 [4,3,0] 4 [4,3,0] 4 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T03:37:48.051461+0000 0 0 2024-04-12T16:25:24.949 INFO:tasks.workunit.client.0.smithi137.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:24:55.483262+0000 0'0 58:111 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-13T23:07:30.834467+0000 0 0 2024-04-12T16:25:24.949 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-04-12T16:25:24.949 INFO:tasks.workunit.client.0.smithi137.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-12T16:25:24.949 INFO:tasks.workunit.client.0.smithi137.stderr:dumped pgs 2024-04-12T16:25:24.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:563: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-04-12T16:25:24.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:25:24.965 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:25:24.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:25:24.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:25:24.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:25:25.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:25:25.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: sleep 5 2024-04-12T16:25:30.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:566: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-04-12T16:25:30.677 INFO:tasks.workunit.client.0.smithi137.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-12T16:25:30.677 INFO:tasks.workunit.client.0.smithi137.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-04-12T16:25:26.602975+0000 52'301 61:913 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-04-12T16:23:09.955422+0000 0'0 2024-04-12T16:23:09.955422+0000 0 0 no scrub is scheduled 0 0 2024-04-12T16:25:30.677 INFO:tasks.workunit.client.0.smithi137.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-12T16:25:25.968847+0000 0'0 61:65 [4,1] 4 [4,1] 4 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T01:13:26.135052+0000 0 0 2024-04-12T16:25:30.677 INFO:tasks.workunit.client.0.smithi137.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.909110+0000 0'0 59:129 [0,4,1] 0 [0,4,1] 0 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T00:07:12.671207+0000 0 0 2024-04-12T16:25:30.678 INFO:tasks.workunit.client.0.smithi137.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:23:09.398019+0000 0'0 61:49 [4,3,0] 4 [4,3,0] 4 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 periodic scrub scheduled @ 2024-04-14T03:37:48.051461+0000 0 0 2024-04-12T16:25:30.678 INFO:tasks.workunit.client.0.smithi137.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-12T16:25:26.504915+0000 0'0 61:119 [1,0] 1 [1,0] 1 0'0 2024-04-12T16:19:58.032308+0000 0'0 2024-04-12T16:19:58.032308+0000 0 0 no scrub is scheduled 0 0 2024-04-12T16:25:30.678 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-04-12T16:25:30.678 INFO:tasks.workunit.client.0.smithi137.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-12T16:25:30.678 INFO:tasks.workunit.client.0.smithi137.stderr:dumped pgs 2024-04-12T16:25:30.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:25:30.849 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "num_missing": 1, 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "num_unfound": 1, 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": [ 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "oid": { 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "oid": "obj250", 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "key": "", 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "snapid": -2, 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": 2249616407, 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "max": 0, 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 2, 2024-04-12T16:25:30.850 INFO:tasks.workunit.client.0.smithi137.stdout: "namespace": "" 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "need": "52'251", 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "have": "0'0", 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "flags": "none", 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "locations": [ 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "NotRecovering", 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "available_might_have_unfound": true, 2024-04-12T16:25:30.851 INFO:tasks.workunit.client.0.smithi137.stdout: "might_have_unfound": [ 2024-04-12T16:25:30.852 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:30.852 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "2(4)", 2024-04-12T16:25:30.852 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "osd is down" 2024-04-12T16:25:30.852 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:30.852 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:30.852 INFO:tasks.workunit.client.0.smithi137.stdout: "more": false 2024-04-12T16:25:30.852 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-04-12T16:25:30.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-04-12T16:25:31.025 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_trimq": "[]", 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_trimq_len": 0, 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 61, 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:25:31.026 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_recovery_backfill": [ 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: "1(1)", 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: "3(0)", 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: "info": { 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s0", 2024-04-12T16:25:31.027 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "0", 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "52'301", 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "52'250", 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "52'200", 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 301, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 44, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 44, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 61, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 59, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 45, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 44, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:25:31.028 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 59, 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 59, 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 44, 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "52'301", 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 914, 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 61, 2024-04-12T16:25:31.029 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:25:26.602975+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:25:26.602975+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:25:26.602975+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:25:26.602975+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:23:18.834622+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:25:26.573754+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:25:26.573754+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:25:26.602975+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:25:26.287135+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:25:26.286638+0000", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 59, 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "52'200", 2024-04-12T16:25:31.030 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "52'200", 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 44, 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 45, 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:25:31.031 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 101, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 200, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 101, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:25:31.032 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "no scrub is scheduled", 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 1232896, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 301, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 1505, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 1, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 1, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 303, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 1, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 301, 2024-04-12T16:25:31.033 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 301, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 1204, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:25:31.034 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:25:31.035 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.036 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [ 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [ 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 300 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.037 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),4(2)", 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 1 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 61, 2024-04-12T16:25:31.038 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "peer_info": [ 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "peer": "0(3)", 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s3", 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "3", 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "52'301", 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "52'301", 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "52'200", 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 301, 2024-04-12T16:25:31.039 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 44, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 44, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 61, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 59, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 45, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 44, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 59, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 59, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 44, 2024-04-12T16:25:31.040 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "52'301", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 903, 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 59, 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized+degraded", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.041 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:23:18.834622+0000", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:23:20.742668+0000", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:23:20.742364+0000", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 59, 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "52'200", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "52'200", 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 44, 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 45, 2024-04-12T16:25:31.042 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 101, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 200, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 101, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:25:31.043 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-04-13T16:32:54.721553+0000", 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 1232896, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 301, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 1505, 2024-04-12T16:25:31.044 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 301, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 301, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 301, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 1204, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-04-12T16:25:31.045 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:25:31.046 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.047 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [ 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: "3(0)", 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: "1(1)", 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [ 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.048 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 301 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 61, 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:25:31.049 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "peer": "1(1)", 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s1", 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "1", 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "52'301", 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "52'301", 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "52'200", 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 301, 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:25:31.050 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 44, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 44, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 61, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 59, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 45, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 44, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 59, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 59, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 44, 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:25:31.051 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "52'301", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 334, 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 52, 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized+degraded", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:23:18.834622+0000", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.052 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:23:20.742668+0000", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:23:20.742364+0000", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 59, 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "52'200", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "52'200", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 44, 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 45, 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:25:31.053 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 101, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 200, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 101, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:25:31.054 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-04-13T16:32:54.721553+0000", 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 1232896, 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 301, 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 1505, 2024-04-12T16:25:31.055 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 1, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 301, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 301, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 301, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 1204, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:25:31.056 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:25:31.057 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.058 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [ 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: "3(0)", 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: "1(1)", 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.059 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [ 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 301 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:25:31.060 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 61, 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "peer": "4(2)", 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s2", 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "2", 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "52'301", 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "52'301", 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "52'200", 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 301, 2024-04-12T16:25:31.061 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 44, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 44, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 61, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 59, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 45, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 44, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 59, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 59, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 44, 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:25:31.062 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "52'301", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 334, 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 52, 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized+degraded", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.063 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:23:18.834622+0000", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:23:20.978499+0000", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:24:47.068484+0000", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:23:20.742668+0000", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:23:20.742364+0000", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 59, 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "52'200", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "52'200", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 44, 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 45, 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:25:31.064 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:23:09.955422+0000", 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 101, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 200, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 101, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:25:31.065 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-04-13T16:32:54.721553+0000", 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 1232896, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 301, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 1505, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-04-12T16:25:31.066 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 301, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 301, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 301, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 1204, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-04-12T16:25:31.067 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:25:31.068 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:25:31.069 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: 2147483647 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [ 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: "3(0)", 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: "1(1)", 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [ 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 301 2024-04-12T16:25:31.070 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 61, 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:25:31.071 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: "recovery_state": [ 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "Started/Primary/Active", 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: "enter_time": "2024-04-12T16:25:26.286714+0000", 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: "might_have_unfound": [ 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "0(3)", 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "already probed" 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "1(1)", 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "already probed" 2024-04-12T16:25:31.072 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "2(4)", 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "osd is down" 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "4(2)", 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "already probed" 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "recovery_progress": { 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "backfill_targets": [], 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "waiting_on_backfill": [], 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill_started": "MIN", 2024-04-12T16:25:31.073 INFO:tasks.workunit.client.0.smithi137.stdout: "backfill_info": { 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "begin": "MIN", 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "end": "MIN", 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": [] 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "peer_backfill_info": [], 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "backfills_in_flight": [], 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "recovering": [], 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_backend": { 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "recovery_ops": [], 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "read_ops": [] 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:25:31.074 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "Started", 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "enter_time": "2024-04-12T16:25:25.416670+0000" 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "scrubber": { 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "active": false, 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "must_scrub": false, 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "must_deep_scrub": false, 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "must_repair": false, 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "need_auto": false, 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_reg_stamp": "2024-04-13T16:32:54.721553+0000", 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "schedule": "no scrub is scheduled" 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout: "agent_state": {} 2024-04-12T16:25:31.075 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-04-12T16:25:31.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:25:31.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: grep -q obj250 2024-04-12T16:25:31.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:25:31.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-04-12T16:25:31.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: check=true 2024-04-12T16:25:31.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: test true == true 2024-04-12T16:25:31.346 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:25:31.346 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-04-12T16:25:31.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-04-12T16:25:31.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: check='osd is down' 2024-04-12T16:25:31.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-04-12T16:25:31.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:25:31.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-04-12T16:25:31.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-04-12T16:25:31.655 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: check='2(4)' 2024-04-12T16:25:31.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-04-12T16:25:31.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:581: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-04-12T16:25:31.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T16:25:31.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:25:31.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T16:25:31.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T16:25:31.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:25:31.658 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:25:31.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-12T16:25:31.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:25:31.659 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-04-12T16:25:31.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-12T16:25:31.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T16:25:31.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T16:25:31.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T16:25:31.688 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:25:31.685+0000 7f25c0db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:25:31.690 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:25:31.689+0000 7f25c0db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:25:31.691 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:25:31.689+0000 7f25c0db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:25:32.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-12T16:25:32.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:25:32.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:25:32.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:25:32.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:25:32.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:25:32.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:25:32.049 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:25:32.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:25:32.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:25:32.238 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:25:32.237+0000 7f25c0db27c0 -1 Falling back to public interface 2024-04-12T16:25:32.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:25:33.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:25:33.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:25:33.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:25:33.556 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:25:33.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:25:33.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:25:33.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:25:34.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:25:34.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:25:34.977 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:25:34.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:25:34.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:25:34.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:25:35.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:25:35.483 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:25:35.481+0000 7f25c0db27c0 -1 osd.2 58 log_to_monitors true 2024-04-12T16:25:36.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:25:36.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:25:36.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:25:36.395 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:25:36.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:25:36.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:25:37.005 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 63 up_thru 19 down_at 59 last_clean_interval [54,58) [v2:127.0.0.1:6818/4259610799,v1:127.0.0.1:6819/4259610799] [v2:127.0.0.1:6820/4259610799,v1:127.0.0.1:6821/4259610799] exists,up 34e44825-6c4a-4e29-8b1b-89c0e685320a 2024-04-12T16:25:37.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:25:37.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:25:37.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:25:37.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-04-12T16:25:42.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:585: TEST_ec_backfill_unfound: test 124 = 124 2024-04-12T16:25:42.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-12T16:25:42.283 INFO:tasks.workunit.client.0.smithi137.stderr:pg has no unfound objects 2024-04-12T16:25:42.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_backfill_unfound: wait_for_clean 2024-04-12T16:25:42.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:25:42.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:25:42.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:25:42.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:25:42.304 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:25:42.304 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:25:42.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:25:42.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:25:42.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:25:42.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:25:42.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:25:42.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:25:42.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:25:42.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:25:42.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:25:42.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:25:42.987 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:25:42.987 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:25:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:25:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:4' 2024-04-12T16:25:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:25:42.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:25:42.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:25:43.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803845 2024-04-12T16:25:43.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803845 2024-04-12T16:25:43.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845' 2024-04-12T16:25:43.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:25:43.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:25:43.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574908 2024-04-12T16:25:43.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574908 2024-04-12T16:25:43.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908' 2024-04-12T16:25:43.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:25:43.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:25:43.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939652 2024-04-12T16:25:43.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939652 2024-04-12T16:25:43.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-270582939652' 2024-04-12T16:25:43.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:25:43.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:25:43.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084339 2024-04-12T16:25:43.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084339 2024-04-12T16:25:43.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-270582939652 3-120259084339' 2024-04-12T16:25:43.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:25:43.651 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:25:43.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757290 2024-04-12T16:25:43.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757290 2024-04-12T16:25:43.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-270582939652 3-120259084339 4-163208757290' 2024-04-12T16:25:43.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:25:43.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803845 2024-04-12T16:25:43.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:25:43.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:25:43.803 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803845 2024-04-12T16:25:43.803 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:25:43.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803845 2024-04-12T16:25:43.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803845' 2024-04-12T16:25:43.804 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803845 2024-04-12T16:25:43.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:25:44.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803843 -lt 25769803845 2024-04-12T16:25:44.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T16:25:45.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T16:25:45.207 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:25:45.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803845 -lt 25769803845 2024-04-12T16:25:45.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:25:45.911 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:25:45.911 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574908 2024-04-12T16:25:45.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:25:45.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574908 2024-04-12T16:25:45.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:25:45.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574908 2024-04-12T16:25:45.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574908' 2024-04-12T16:25:45.915 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574908 2024-04-12T16:25:45.915 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:25:46.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574909 -lt 55834574908 2024-04-12T16:25:46.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:25:46.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-270582939652 2024-04-12T16:25:46.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:25:46.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:25:46.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-270582939652 2024-04-12T16:25:46.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:25:46.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939652 2024-04-12T16:25:46.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 270582939652' 2024-04-12T16:25:46.414 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 270582939652 2024-04-12T16:25:46.414 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:25:46.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939653 -lt 270582939652 2024-04-12T16:25:46.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:25:46.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084339 2024-04-12T16:25:46.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:25:46.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:25:46.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084339 2024-04-12T16:25:46.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:25:46.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084339 2024-04-12T16:25:46.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084339' 2024-04-12T16:25:46.998 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 120259084339 2024-04-12T16:25:46.998 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:25:47.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084340 -lt 120259084339 2024-04-12T16:25:47.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:25:47.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757290 2024-04-12T16:25:47.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:25:47.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:25:47.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757290 2024-04-12T16:25:47.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:25:47.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757290 2024-04-12T16:25:47.475 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 163208757290 2024-04-12T16:25:47.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757290' 2024-04-12T16:25:47.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:25:48.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757290 -lt 163208757290 2024-04-12T16:25:48.018 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:25:48.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:25:48.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:25:48.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:25:48.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:25:48.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:25:48.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:25:48.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:25:48.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:25:48.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:25:48.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:25:49.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T16:25:49.012 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:25:49.012 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:25:49.012 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:25:49.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-12T16:25:49.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-12T16:25:49.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-12T16:25:49.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-12T16:25:49.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-12T16:25:49.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-12T16:25:49.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-12T16:25:49.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:25:49.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:25:49.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:25:49.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:25:49.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:25:49.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:25:49.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:25:50.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T16:25:50.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:25:50.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:25:50.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:25:50.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-12T16:25:50.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-04-12T16:25:50.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-12T16:25:50.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-12T16:25:50.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-12T16:25:50.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-12T16:25:50.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-12T16:25:50.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-12T16:25:50.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-12T16:25:51.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-12T16:25:51.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-12T16:25:51.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-12T16:25:51.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-12T16:25:51.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-12T16:25:51.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-12T16:25:51.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:25:51.422 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:25:51.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:25:51.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:25:51.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:25:51.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:25:51.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:25:51.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:25:51.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:25:51.825 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:25:51.825 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:25:52.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:25:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:25:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:25:52.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: seq 1 300 2024-04-12T16:25:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-04-12T16:25:52.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-04-12T16:25:52.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-04-12T16:25:52.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-04-12T16:25:52.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-04-12T16:25:52.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-04-12T16:25:52.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-04-12T16:25:52.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-04-12T16:25:52.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-04-12T16:25:52.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-04-12T16:25:52.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-04-12T16:25:52.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-04-12T16:25:52.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-04-12T16:25:52.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-04-12T16:25:52.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-04-12T16:25:52.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-04-12T16:25:52.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-04-12T16:25:52.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-04-12T16:25:52.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-04-12T16:25:52.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:52.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:52.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-04-12T16:25:52.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-04-12T16:25:53.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-04-12T16:25:53.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-04-12T16:25:53.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-04-12T16:25:53.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-04-12T16:25:53.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-04-12T16:25:53.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-04-12T16:25:53.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-04-12T16:25:53.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-04-12T16:25:53.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-04-12T16:25:53.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-04-12T16:25:53.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-04-12T16:25:53.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-04-12T16:25:53.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-04-12T16:25:53.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-04-12T16:25:53.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-04-12T16:25:53.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-04-12T16:25:53.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-04-12T16:25:53.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-04-12T16:25:53.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-04-12T16:25:53.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-04-12T16:25:53.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-04-12T16:25:53.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-04-12T16:25:53.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-04-12T16:25:53.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-04-12T16:25:53.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-04-12T16:25:53.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-04-12T16:25:53.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-04-12T16:25:53.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-04-12T16:25:53.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-04-12T16:25:53.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-04-12T16:25:53.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-04-12T16:25:53.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-04-12T16:25:53.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-04-12T16:25:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-04-12T16:25:54.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-04-12T16:25:54.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-04-12T16:25:54.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-04-12T16:25:54.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-04-12T16:25:54.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-04-12T16:25:54.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-04-12T16:25:54.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-04-12T16:25:54.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-04-12T16:25:54.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-04-12T16:25:54.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-04-12T16:25:54.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-04-12T16:25:54.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-04-12T16:25:54.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-04-12T16:25:54.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-04-12T16:25:54.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-04-12T16:25:54.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-04-12T16:25:54.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-04-12T16:25:54.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-04-12T16:25:54.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-04-12T16:25:54.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-04-12T16:25:54.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-04-12T16:25:54.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-04-12T16:25:54.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-04-12T16:25:54.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-04-12T16:25:54.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-04-12T16:25:54.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-04-12T16:25:54.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-04-12T16:25:54.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-04-12T16:25:54.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-04-12T16:25:54.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-04-12T16:25:54.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-04-12T16:25:54.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-04-12T16:25:54.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:54.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:54.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-04-12T16:25:54.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-04-12T16:25:55.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-04-12T16:25:55.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-04-12T16:25:55.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-04-12T16:25:55.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-04-12T16:25:55.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-04-12T16:25:55.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-04-12T16:25:55.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-04-12T16:25:55.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-04-12T16:25:55.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-04-12T16:25:55.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-04-12T16:25:55.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-04-12T16:25:55.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-04-12T16:25:55.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-04-12T16:25:55.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-04-12T16:25:55.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-04-12T16:25:55.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-04-12T16:25:55.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-04-12T16:25:55.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-04-12T16:25:55.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-04-12T16:25:55.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-04-12T16:25:55.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-04-12T16:25:55.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-04-12T16:25:55.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-04-12T16:25:55.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-04-12T16:25:55.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-04-12T16:25:55.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-04-12T16:25:55.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-04-12T16:25:55.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-04-12T16:25:55.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-04-12T16:25:55.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-04-12T16:25:55.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-04-12T16:25:55.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-04-12T16:25:55.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-04-12T16:25:55.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-04-12T16:25:55.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-04-12T16:25:56.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-04-12T16:25:56.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-04-12T16:25:56.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-04-12T16:25:56.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-04-12T16:25:56.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-04-12T16:25:56.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-04-12T16:25:56.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-04-12T16:25:56.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-04-12T16:25:56.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-04-12T16:25:56.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-04-12T16:25:56.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-04-12T16:25:56.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-04-12T16:25:56.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-04-12T16:25:56.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-04-12T16:25:56.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-04-12T16:25:56.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-04-12T16:25:56.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-04-12T16:25:56.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-04-12T16:25:56.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-04-12T16:25:56.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-04-12T16:25:56.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-04-12T16:25:56.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-04-12T16:25:56.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-04-12T16:25:56.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-04-12T16:25:56.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-04-12T16:25:56.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-04-12T16:25:56.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-04-12T16:25:56.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-04-12T16:25:56.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-04-12T16:25:56.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-04-12T16:25:56.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-04-12T16:25:56.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-04-12T16:25:56.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:56.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:56.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-04-12T16:25:56.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-04-12T16:25:57.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-04-12T16:25:57.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-04-12T16:25:57.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-04-12T16:25:57.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-04-12T16:25:57.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-04-12T16:25:57.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-04-12T16:25:57.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-04-12T16:25:57.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-04-12T16:25:57.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-04-12T16:25:57.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-04-12T16:25:57.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-04-12T16:25:57.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-04-12T16:25:57.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-04-12T16:25:57.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-04-12T16:25:57.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-04-12T16:25:57.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-04-12T16:25:57.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-04-12T16:25:57.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-04-12T16:25:57.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-04-12T16:25:57.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-04-12T16:25:57.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-04-12T16:25:57.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-04-12T16:25:57.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-04-12T16:25:57.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-04-12T16:25:57.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-04-12T16:25:57.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-04-12T16:25:57.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-04-12T16:25:57.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-04-12T16:25:57.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-04-12T16:25:57.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-04-12T16:25:57.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-04-12T16:25:57.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-04-12T16:25:57.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-04-12T16:25:57.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:57.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:57.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-04-12T16:25:57.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-04-12T16:25:58.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-04-12T16:25:58.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-04-12T16:25:58.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-04-12T16:25:58.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-04-12T16:25:58.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-04-12T16:25:58.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-04-12T16:25:58.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-04-12T16:25:58.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-04-12T16:25:58.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-04-12T16:25:58.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-04-12T16:25:58.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-04-12T16:25:58.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-04-12T16:25:58.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-04-12T16:25:58.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-04-12T16:25:58.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-04-12T16:25:58.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-04-12T16:25:58.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-04-12T16:25:58.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-04-12T16:25:58.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-04-12T16:25:58.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-04-12T16:25:58.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-04-12T16:25:58.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-04-12T16:25:58.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-04-12T16:25:58.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-04-12T16:25:58.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-04-12T16:25:58.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-04-12T16:25:58.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-04-12T16:25:58.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-04-12T16:25:58.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-04-12T16:25:58.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-04-12T16:25:58.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-04-12T16:25:58.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-04-12T16:25:58.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:58.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:58.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-04-12T16:25:58.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-04-12T16:25:59.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-04-12T16:25:59.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-04-12T16:25:59.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-04-12T16:25:59.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-04-12T16:25:59.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-04-12T16:25:59.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-04-12T16:25:59.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-04-12T16:25:59.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-04-12T16:25:59.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-04-12T16:25:59.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-04-12T16:25:59.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-04-12T16:25:59.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-04-12T16:25:59.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-04-12T16:25:59.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-04-12T16:25:59.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-04-12T16:25:59.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-04-12T16:25:59.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-04-12T16:25:59.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-04-12T16:25:59.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-04-12T16:25:59.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-04-12T16:25:59.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-04-12T16:25:59.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-04-12T16:25:59.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-04-12T16:25:59.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-04-12T16:25:59.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-04-12T16:25:59.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-04-12T16:25:59.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-04-12T16:25:59.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-04-12T16:25:59.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-04-12T16:25:59.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-04-12T16:25:59.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.923 INFO:tasks.workunit.client.0.smithi137.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-04-12T16:25:59.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-04-12T16:25:59.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-04-12T16:25:59.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-04-12T16:25:59.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:25:59.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:25:59.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-04-12T16:25:59.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-04-12T16:26:00.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-04-12T16:26:00.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-04-12T16:26:00.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-04-12T16:26:00.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-04-12T16:26:00.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-04-12T16:26:00.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-04-12T16:26:00.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-04-12T16:26:00.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-04-12T16:26:00.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-04-12T16:26:00.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-04-12T16:26:00.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-04-12T16:26:00.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-04-12T16:26:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-04-12T16:26:00.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-04-12T16:26:00.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-04-12T16:26:00.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-04-12T16:26:00.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-04-12T16:26:00.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-04-12T16:26:00.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-04-12T16:26:00.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-04-12T16:26:00.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-04-12T16:26:00.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-04-12T16:26:00.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-04-12T16:26:00.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-04-12T16:26:00.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-04-12T16:26:00.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-04-12T16:26:00.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-04-12T16:26:00.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-04-12T16:26:00.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-04-12T16:26:00.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-04-12T16:26:00.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-04-12T16:26:00.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-04-12T16:26:00.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:00.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:00.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-04-12T16:26:00.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-04-12T16:26:01.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-04-12T16:26:01.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-04-12T16:26:01.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-04-12T16:26:01.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-04-12T16:26:01.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-04-12T16:26:01.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-04-12T16:26:01.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-04-12T16:26:01.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-04-12T16:26:01.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-04-12T16:26:01.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-04-12T16:26:01.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-04-12T16:26:01.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-04-12T16:26:01.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:26:01.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-04-12T16:26:01.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-04-12T16:26:01.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:26:01.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-04-12T16:26:01.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:26:01.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T16:26:01.946 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T16:26:01.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-04-12T16:26:02.961 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile myprofile does not exist 2024-04-12T16:26:02.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-04-12T16:26:02.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:26:02.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:26:02.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:26:02.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:26:02.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:26:02.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:26:02.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:26:02.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:26:03.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:26:03.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:26:03.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:26:03.297 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:26:03.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:26:03.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:26:03.298 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:26:03.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:26:03.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:26:03.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:26:03.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:26:03.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:26:03.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:26:03.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:26:03.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:26:03.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:03.366 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:03.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:26:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:26:03.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:26:03.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:26:03.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:26:03.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:26:03.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:26:03.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:26:03.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:26:03.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:26:03.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:26:03.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:26:03.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:26:03.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:26:03.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:26:03.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:26:03.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:26:03.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:26:03.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:03.378 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:03.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:26:03.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:26:03.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:26:03.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T16:26:03.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:26:03.381 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:03.381 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:03.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T16:26:03.383 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:26:03.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:26:03.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:26:03.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T16:26:03.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T16:26:03.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T16:26:03.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:26:03.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:26:03.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:26:03.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T16:26:03.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T16:26:03.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:26:03.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:26:03.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:26:03.855 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:26:03.856 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:03.856 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:03.856 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:26:03.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:26:04.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:26:04.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:26:04.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:26:04.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:26:04.067 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:26:04.068 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:26:04.068 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:26:04.068 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:26:04.068 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:26:04.069 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:26:04.069 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:04.069 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:04.069 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:26:04.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:26:04.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T16:26:04.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:26:04.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:26:04.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:26:04.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:26:04.184 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:26:04.184 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:26:04.185 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:26:04.185 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:26:04.185 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:26:04.185 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:04.185 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:04.185 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:26:04.186 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:26:04.186 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T16:26:04.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T16:26:04.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T16:26:04.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:26:04.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:26:04.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:26:04.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T16:26:04.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:26:05.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:26:05.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:26:05.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:26:05.205 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:26:05.206 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:05.206 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:05.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:26:05.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:26:05.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:26:05.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T16:26:05.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:26:05.578 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:26:05.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:26:06.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T16:26:06.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:26:06.598 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:26:06.599 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:26:06.599 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:06.599 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:06.599 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:26:06.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T16:26:06.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T16:26:06.699 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:26:06.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T16:26:06.714 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:26:03.897+0000 7f3e34836c80 0 load: jerasure load: lrc 2024-04-12T16:26:06.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-04-12T16:26:06.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-04-12T16:26:06.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_ec_object_attr_read_error: local objname=myobject 2024-04-12T16:26:06.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:362: TEST_ec_object_attr_read_error: setup_osds 7 2024-04-12T16:26:06.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-04-12T16:26:06.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T16:26:06.715 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-04-12T16:26:06.717 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-04-12T16:26:06.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:26:06.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:26:06.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:26:06.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:26:06.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=314dc9fd-1c91-40f1-937a-18f473771ae6 2024-04-12T16:26:06.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 314dc9fd-1c91-40f1-937a-18f473771ae6' 2024-04-12T16:26:06.725 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 314dc9fd-1c91-40f1-937a-18f473771ae6 2024-04-12T16:26:06.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:26:06.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeYBlmBFI5LBAAzkIAOIJgRuN87m31fnJPYA== 2024-04-12T16:26:06.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeYBlmBFI5LBAAzkIAOIJgRuN87m31fnJPYA=="}' 2024-04-12T16:26:06.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 314dc9fd-1c91-40f1-937a-18f473771ae6 -i td/test-erasure-eio/0/new.json 2024-04-12T16:26:07.070 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:26:07.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T16:26:07.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCeYBlmBFI5LBAAzkIAOIJgRuN87m31fnJPYA== --osd-uuid 314dc9fd-1c91-40f1-937a-18f473771ae6 2024-04-12T16:26:07.125 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:07.125+0000 7f449ef9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:07.147 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:07.145+0000 7f449ef9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:07.148 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:07.145+0000 7f449ef9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:07.149 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:07.149+0000 7f449ef9d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:26:07.149 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:07.149+0000 7f449ef9d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:26:07.149 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:07.149+0000 7f449ef9d7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:26:07.187 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:07.185+0000 7f449ef9d7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T16:26:13.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T16:26:13.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:26:13.844 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:26:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:26:13.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:26:14.387 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:26:14.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:26:14.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:26:14.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:26:14.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:26:14.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:26:14.419 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:14.417+0000 7fc2a9b967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:14.445 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:14.441+0000 7fc2a9b967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:14.523 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:14.445+0000 7fc2a9b967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:14.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:26:14.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:26:14.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:26:14.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:26:14.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:26:14.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:14.834 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:26:14.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:26:14.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:14.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:15.019 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:15.017+0000 7fc2a9b967c0 -1 Falling back to public interface 2024-04-12T16:26:15.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:16.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:16.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:16.281 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:26:16.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:26:16.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:16.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:16.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:17.678 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:17.677+0000 7fc2a9b967c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:26:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:17.686 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:26:17.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:26:17.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:17.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:18.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:19.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:19.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:19.294 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:26:19.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:26:19.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:19.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:19.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:20.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:20.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:20.735 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:26:20.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:26:20.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:20.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:21.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:22.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:22.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:22.148 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:26:22.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:26:22.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:22.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:22.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:23.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:23.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:23.597 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:26:23.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:26:23.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:23.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:24.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:25.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:25.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:25.026 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:26:25.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:26:25.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:25.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:25.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:26.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:26.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:26.565 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:26:26.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:26:26.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:26.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:27.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:28.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:28.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:28.099 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:26:28.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:26:28.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:28.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:28.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:29.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:29.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:29.543 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:26:29.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:26:29.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:29.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:30.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:31.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:31.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:31.075 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:26:31.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:26:31.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:31.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:32.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:33.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:33.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:33.135 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:26:33.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:26:33.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:33.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:34.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:35.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:35.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:35.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:26:35.043 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:26:35.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:35.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:35.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:36.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:36.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:36.902 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:26:36.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:26:36.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:36.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:37.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:38.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:38.727 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:26:38.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:38.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:26:38.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:38.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:39.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:40.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:40.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:40.927 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:26:40.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:26:40.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:40.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:41.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:42.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:42.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:42.844 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:26:42.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:26:42.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:42.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:26:43.401 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4095193335,v1:127.0.0.1:6803/4095193335] [v2:127.0.0.1:6804/4095193335,v1:127.0.0.1:6805/4095193335] exists,up 314dc9fd-1c91-40f1-937a-18f473771ae6 2024-04-12T16:26:43.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:26:43.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:26:43.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:26:43.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:26:43.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T16:26:43.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:26:43.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:26:43.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:26:43.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:26:43.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:26:43.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:26:43.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:26:43.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:26:43.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:26:43.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T16:26:43.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:26:43.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eacd6c5d-383c-47d9-9a96-8087faf4ed83 2024-04-12T16:26:43.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 eacd6c5d-383c-47d9-9a96-8087faf4ed83' 2024-04-12T16:26:43.410 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 eacd6c5d-383c-47d9-9a96-8087faf4ed83 2024-04-12T16:26:43.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:26:43.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDYBlmQmecGRAAD71wykpYfTnVsZI+iBuXSw== 2024-04-12T16:26:43.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDYBlmQmecGRAAD71wykpYfTnVsZI+iBuXSw=="}' 2024-04-12T16:26:43.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eacd6c5d-383c-47d9-9a96-8087faf4ed83 -i td/test-erasure-eio/1/new.json 2024-04-12T16:26:44.073 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:26:44.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T16:26:44.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDYBlmQmecGRAAD71wykpYfTnVsZI+iBuXSw== --osd-uuid eacd6c5d-383c-47d9-9a96-8087faf4ed83 2024-04-12T16:26:44.126 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:44.125+0000 7fa1dd6667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:44.146 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:44.145+0000 7fa1dd6667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:44.147 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:44.145+0000 7fa1dd6667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:44.148 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:44.145+0000 7fa1dd6667c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:26:44.148 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:44.145+0000 7fa1dd6667c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:26:44.148 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:44.145+0000 7fa1dd6667c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:26:44.193 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:44.193+0000 7fa1dd6667c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T16:26:50.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T16:26:50.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:26:50.868 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:26:50.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:26:50.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:26:51.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:26:51.410 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:26:51.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:26:51.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:26:51.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:26:51.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:26:51.443 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:51.441+0000 7f02db2077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:51.445+0000 7f02db2077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:51.446 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:51.445+0000 7f02db2077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:26:51.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:26:51.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:26:51.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:26:51.872 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:26:51.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:26:51.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:26:51.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:51.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:26:51.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:51.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:26:51.998 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:51.997+0000 7f02db2077c0 -1 Falling back to public interface 2024-04-12T16:26:52.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:53.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:53.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:53.268 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:26:53.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:26:53.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:53.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:26:53.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:54.584 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:26:54.581+0000 7f02db2077c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:26:54.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:54.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:54.676 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:26:54.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:26:54.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:54.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:26:55.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:56.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:56.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:56.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:26:56.347 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:26:56.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:56.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:26:56.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:57.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:57.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:57.874 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:26:57.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:26:57.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:57.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:26:58.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:26:59.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:26:59.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:26:59.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:26:59.464 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:26:59.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:26:59.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:26:59.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:00.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:00.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:00.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:27:00.911 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:27:00.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:00.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:01.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:02.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:02.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:02.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:27:02.525 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:27:02.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:02.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:03.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:04.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:04.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:04.125 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:27:04.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:27:04.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:04.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:04.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:05.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:05.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:05.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:27:05.664 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:27:05.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:05.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:06.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:07.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:07.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:07.146 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:27:07.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:27:07.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:07.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:07.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:08.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:08.629 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:27:08.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:08.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:27:08.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:08.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:09.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:10.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:10.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:10.140 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:27:10.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:27:10.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:10.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:11.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:12.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:12.392 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:27:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:27:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:12.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:13.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:13.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:13.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:27:13.938 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:27:13.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:13.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:14.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:15.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:15.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:15.799 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:27:15.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:27:15.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:15.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:16.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:17.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:17.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:17.377 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:27:17.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:27:17.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:17.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:18.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:19.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:19.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:19.200 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:27:19.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:27:19.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:19.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:19.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:20.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:21.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:21.181 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:27:21.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:27:21.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:21.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:27:21.471 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3368690332,v1:127.0.0.1:6811/3368690332] [v2:127.0.0.1:6812/3368690332,v1:127.0.0.1:6813/3368690332] exists,up eacd6c5d-383c-47d9-9a96-8087faf4ed83 2024-04-12T16:27:21.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:27:21.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:27:21.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:27:21.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:27:21.478 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:27:21.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e9db926-c2d4-4bc1-a1ed-3943d91428d9 2024-04-12T16:27:21.478 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 2e9db926-c2d4-4bc1-a1ed-3943d91428d9 2024-04-12T16:27:21.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2e9db926-c2d4-4bc1-a1ed-3943d91428d9' 2024-04-12T16:27:21.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:27:21.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpYBlm1rqzHRAARme5ZfP+heK+fqnjp0EKGQ== 2024-04-12T16:27:21.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpYBlm1rqzHRAARme5ZfP+heK+fqnjp0EKGQ=="}' 2024-04-12T16:27:21.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e9db926-c2d4-4bc1-a1ed-3943d91428d9 -i td/test-erasure-eio/2/new.json 2024-04-12T16:27:22.014 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:27:22.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T16:27:22.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpYBlm1rqzHRAARme5ZfP+heK+fqnjp0EKGQ== --osd-uuid 2e9db926-c2d4-4bc1-a1ed-3943d91428d9 2024-04-12T16:27:22.064 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:22.061+0000 7f14deac57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:27:22.141 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:22.137+0000 7f14deac57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:27:22.142 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:22.141+0000 7f14deac57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:27:22.143 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:22.141+0000 7f14deac57c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:27:22.143 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:22.141+0000 7f14deac57c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:27:22.143 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:22.141+0000 7f14deac57c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:27:22.264 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:22.261+0000 7f14deac57c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T16:27:29.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T16:27:29.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:27:29.219 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:27:29.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:27:29.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:27:29.782 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:27:29.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:27:29.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:27:29.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:27:29.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:27:29.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:27:29.813 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:29.813+0000 7ff0022187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:27:29.815 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:29.813+0000 7ff0022187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:27:29.817 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:29.813+0000 7ff0022187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:27:30.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:27:30.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:27:30.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:27:30.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:27:30.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:27:30.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:30.288 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:27:30.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:27:30.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:30.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:30.383 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:30.381+0000 7ff0022187c0 -1 Falling back to public interface 2024-04-12T16:27:30.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:31.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:31.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:31.733 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:27:31.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:27:31.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:31.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:32.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:33.040 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:27:33.037+0000 7ff0022187c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:27:33.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:33.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:33.170 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:27:33.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:27:33.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:33.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:33.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:34.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:34.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:34.852 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:27:34.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:27:34.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:34.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:35.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:36.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:36.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:36.339 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:27:36.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:27:36.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:36.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:36.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:37.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:37.875 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:27:37.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:27:37.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:37.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:38.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:39.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:39.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:39.348 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:27:39.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:27:39.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:39.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:39.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:40.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:40.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:40.784 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:27:40.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:27:40.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:40.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:41.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:42.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:42.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:42.259 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:27:42.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:27:42.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:42.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:42.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:43.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:43.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:43.765 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:27:43.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:27:43.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:43.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:44.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:45.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:45.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:45.358 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:27:45.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:27:45.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:45.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:45.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:46.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:46.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:46.799 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:27:46.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:27:46.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:46.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:47.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:48.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:48.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:48.465 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:27:48.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:27:48.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:48.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:49.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:50.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:50.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:50.710 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:27:50.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:27:50.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:50.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:51.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:52.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:52.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:52.509 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:27:52.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:27:52.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:52.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:53.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:54.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:54.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:54.450 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:27:54.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:27:54.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:54.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:55.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:56.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:56.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:56.339 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:27:56.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:27:56.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:56.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:57.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:58.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:58.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:58.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:27:58.287 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:27:58.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:58.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:27:58.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:27:59.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:27:59.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:27:59.772 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:27:59.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:27:59.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:27:59.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:28:00.355 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2703820882,v1:127.0.0.1:6819/2703820882] [v2:127.0.0.1:6820/2703820882,v1:127.0.0.1:6821/2703820882] exists,up 2e9db926-c2d4-4bc1-a1ed-3943d91428d9 2024-04-12T16:28:00.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:28:00.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:28:00.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:28:00.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:28:00.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T16:28:00.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:28:00.357 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:28:00.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:28:00.359 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:28:00.359 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:28:00.359 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:28:00.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:28:00.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:28:00.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:28:00.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:28:00.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:28:00.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:28:00.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:28:00.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=21e95c1f-4e62-4b9a-a28b-b6a24423750b 2024-04-12T16:28:00.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 21e95c1f-4e62-4b9a-a28b-b6a24423750b' 2024-04-12T16:28:00.362 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 21e95c1f-4e62-4b9a-a28b-b6a24423750b 2024-04-12T16:28:00.362 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:28:00.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQYRlmFzfyFhAA2qGRLBqQbIQN2Qxe/ZJoFQ== 2024-04-12T16:28:00.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQYRlmFzfyFhAA2qGRLBqQbIQN2Qxe/ZJoFQ=="}' 2024-04-12T16:28:00.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 21e95c1f-4e62-4b9a-a28b-b6a24423750b -i td/test-erasure-eio/3/new.json 2024-04-12T16:28:01.392 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:28:01.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T16:28:01.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQYRlmFzfyFhAA2qGRLBqQbIQN2Qxe/ZJoFQ== --osd-uuid 21e95c1f-4e62-4b9a-a28b-b6a24423750b 2024-04-12T16:28:01.450 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:01.449+0000 7fe80e1987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:01.519 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:01.517+0000 7fe80e1987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:01.521 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:01.521+0000 7fe80e1987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:01.522 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:01.521+0000 7fe80e1987c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:28:01.522 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:01.521+0000 7fe80e1987c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:28:01.522 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:01.521+0000 7fe80e1987c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:28:01.647 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:01.645+0000 7fe80e1987c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T16:28:08.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T16:28:08.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:28:08.276 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T16:28:08.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T16:28:08.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:28:08.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T16:28:08.849 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:28:08.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:28:08.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:28:08.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:28:08.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:28:08.880 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:08.877+0000 7fa93c5917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:08.905 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:08.905+0000 7fa93c5917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:08.906 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:08.905+0000 7fa93c5917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:09.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T16:28:09.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:28:09.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:28:09.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:28:09.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:28:09.259 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:28:09.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:09.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:28:09.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:09.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:09.459 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:09.457+0000 7fa93c5917c0 -1 Falling back to public interface 2024-04-12T16:28:09.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:10.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:10.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:10.718 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:28:10.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:28:10.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:10.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:11.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:12.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:12.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:12.146 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:28:12.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:28:12.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:12.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:12.160 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:12.157+0000 7fa93c5917c0 -1 osd.3 0 log_to_monitors true 2024-04-12T16:28:12.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:13.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:13.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:13.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:28:13.562 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:28:13.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:13.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:14.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:15.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:15.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:15.248 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:28:15.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:28:15.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:15.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:16.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:16.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:16.942 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:28:16.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:28:16.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:16.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:17.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:18.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:18.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:18.384 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:28:18.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:28:18.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:18.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:18.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:19.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:19.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:19.838 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:28:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:28:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:19.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:20.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:21.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:21.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:21.608 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:28:21.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:28:21.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:21.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:22.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:23.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:23.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:23.254 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:28:23.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:28:23.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:23.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:23.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:24.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:24.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:24.685 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:28:24.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:28:24.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:24.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:25.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:26.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:26.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:26.220 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:28:26.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:28:26.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:26.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:26.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:27.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:27.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:27.711 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:28:27.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:28:27.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:27.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:28.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:29.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:29.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:29.883 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:28:29.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:28:29.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:29.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:30.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:31.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:31.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:31.876 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:28:31.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:28:31.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:31.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:32.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:33.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:33.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:33.739 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:28:33.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:28:33.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:33.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:34.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:35.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:35.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:35.622 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:28:35.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:28:35.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:35.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:36.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:37.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:37.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:37.378 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:28:37.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:28:37.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:37.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:38.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:39.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:39.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:39.339 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:28:39.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:28:39.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:39.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:28:40.004 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3829157797,v1:127.0.0.1:6827/3829157797] [v2:127.0.0.1:6828/3829157797,v1:127.0.0.1:6829/3829157797] exists,up 21e95c1f-4e62-4b9a-a28b-b6a24423750b 2024-04-12T16:28:40.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:28:40.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:28:40.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:28:40.008 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:28:40.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:28:40.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:28:40.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-04-12T16:28:40.010 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:28:40.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d2ac5ef1-2d93-4af9-bcf8-a247129c6279 2024-04-12T16:28:40.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 d2ac5ef1-2d93-4af9-bcf8-a247129c6279' 2024-04-12T16:28:40.011 INFO:tasks.workunit.client.0.smithi137.stdout:add osd4 d2ac5ef1-2d93-4af9-bcf8-a247129c6279 2024-04-12T16:28:40.011 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:28:40.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4YRlmKL0LAhAA4t0uefOZw2a4Z+65DH2Eng== 2024-04-12T16:28:40.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4YRlmKL0LAhAA4t0uefOZw2a4Z+65DH2Eng=="}' 2024-04-12T16:28:40.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d2ac5ef1-2d93-4af9-bcf8-a247129c6279 -i td/test-erasure-eio/4/new.json 2024-04-12T16:28:41.022 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:28:41.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-04-12T16:28:41.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4YRlmKL0LAhAA4t0uefOZw2a4Z+65DH2Eng== --osd-uuid d2ac5ef1-2d93-4af9-bcf8-a247129c6279 2024-04-12T16:28:41.075 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:41.073+0000 7fae3a62f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:41.078 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:41.077+0000 7fae3a62f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:41.079 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:41.077+0000 7fae3a62f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:41.079 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:41.077+0000 7fae3a62f7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:28:41.079 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:41.077+0000 7fae3a62f7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:28:41.079 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:41.077+0000 7fae3a62f7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:28:41.242 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:41.241+0000 7fae3a62f7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-04-12T16:28:47.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-04-12T16:28:47.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:28:47.991 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd4 key to auth repository 2024-04-12T16:28:47.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-12T16:28:47.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:28:48.553 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.4 2024-04-12T16:28:48.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-12T16:28:48.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:28:48.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:28:48.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:28:48.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:28:48.584 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:48.585+0000 7f135fc667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:48.591 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:48.589+0000 7f135fc667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:48.592 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:48.593+0000 7f135fc667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:28:48.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-12T16:28:48.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:28:48.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-12T16:28:48.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:28:48.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:28:48.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:48.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:28:48.978 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:28:48.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:48.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:28:49.142 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:49.141+0000 7f135fc667c0 -1 Falling back to public interface 2024-04-12T16:28:49.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:50.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:50.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:50.380 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:28:50.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:28:50.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:50.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:28:50.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:51.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:51.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:51.791 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:28:51.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:28:51.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:51.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:28:51.832 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:28:51.833+0000 7f135fc667c0 -1 osd.4 0 log_to_monitors true 2024-04-12T16:28:52.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:53.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:53.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:53.344 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:28:53.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:28:53.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:28:53.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:53.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:54.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:54.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:54.988 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:28:54.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:28:54.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:54.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:28:55.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:56.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:56.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:56.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:28:56.526 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:28:56.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:56.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:28:56.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:57.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:57.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:57.957 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:28:57.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:28:57.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:57.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:28:58.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:28:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:28:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:28:59.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:28:59.417 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:28:59.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:28:59.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:28:59.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:00.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:00.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:00.861 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:29:00.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:29:00.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:00.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:01.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:02.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:02.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:02.299 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:29:02.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:29:02.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:02.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:02.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:03.739 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:29:03.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:29:03.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:03.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:04.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:05.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:05.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:05.166 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:29:05.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:29:05.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:05.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:05.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:06.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:06.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:06.664 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:29:06.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:29:06.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:06.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:07.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:08.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:08.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:08.216 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:29:08.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:29:08.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:08.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:09.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:10.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:10.374 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:29:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:29:10.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:10.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:10.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:11.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:11.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:11.968 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:29:11.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:29:11.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:11.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:12.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:13.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:13.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:13.447 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:29:13.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:29:13.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:13.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:14.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:15.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:15.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:15.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:29:15.239 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:29:15.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:15.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:16.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:17.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:17.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:17.145 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:29:17.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:29:17.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:17.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:18.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:19.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:19.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:19.140 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:29:19.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:29:19.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:19.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:29:19.601 INFO:tasks.workunit.client.0.smithi137.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/246456939,v1:127.0.0.1:6835/246456939] [v2:127.0.0.1:6836/246456939,v1:127.0.0.1:6837/246456939] exists,up d2ac5ef1-2d93-4af9-bcf8-a247129c6279 2024-04-12T16:29:19.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:29:19.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:29:19.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:29:19.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:29:19.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-04-12T16:29:19.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:29:19.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:29:19.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:29:19.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:29:19.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:29:19.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:29:19.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:29:19.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:29:19.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:29:19.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-04-12T16:29:19.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:29:19.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b0156a0-f700-4565-8a03-decb441e93f5 2024-04-12T16:29:19.609 INFO:tasks.workunit.client.0.smithi137.stdout:add osd5 4b0156a0-f700-4565-8a03-decb441e93f5 2024-04-12T16:29:19.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 4b0156a0-f700-4565-8a03-decb441e93f5' 2024-04-12T16:29:19.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:29:19.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfYRlm/RmPJRAA9xAZNwGlg7tGUVnlBNBe6g== 2024-04-12T16:29:19.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfYRlm/RmPJRAA9xAZNwGlg7tGUVnlBNBe6g=="}' 2024-04-12T16:29:19.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b0156a0-f700-4565-8a03-decb441e93f5 -i td/test-erasure-eio/5/new.json 2024-04-12T16:29:20.549 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:29:20.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-04-12T16:29:20.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfYRlm/RmPJRAA9xAZNwGlg7tGUVnlBNBe6g== --osd-uuid 4b0156a0-f700-4565-8a03-decb441e93f5 2024-04-12T16:29:20.606 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:20.605+0000 7ff765f907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:20.673 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:20.673+0000 7ff765f907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:20.675 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:20.673+0000 7ff765f907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:20.676 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:20.673+0000 7ff765f907c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:29:20.676 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:20.673+0000 7ff765f907c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:29:20.676 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:20.673+0000 7ff765f907c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:29:20.863 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:20.865+0000 7ff765f907c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-04-12T16:29:27.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-04-12T16:29:27.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:29:27.648 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd5 key to auth repository 2024-04-12T16:29:27.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-12T16:29:27.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:29:28.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-12T16:29:28.243 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.5 2024-04-12T16:29:28.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:29:28.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:29:28.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:29:28.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:29:28.274 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:28.273+0000 7fa50f8a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:28.276 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:28.277+0000 7fa50f8a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:28.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:28.277+0000 7fa50f8a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:28.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-12T16:29:28.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:29:28.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-12T16:29:28.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:29:28.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:29:28.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:28.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:29:28.705 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:29:28.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:28.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:28.838 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:28.837+0000 7fa50f8a87c0 -1 Falling back to public interface 2024-04-12T16:29:29.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:30.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:30.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:30.176 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:29:30.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:29:30.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:30.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:30.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:31.381 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:31.381+0000 7fa50f8a87c0 -1 osd.5 0 log_to_monitors true 2024-04-12T16:29:31.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:31.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:31.623 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:29:31.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:29:31.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:31.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:32.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:33.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:33.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:33.426 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:29:33.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:29:33.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:33.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:34.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:35.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:35.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:35.059 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:29:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:29:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:35.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:35.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:36.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:36.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:36.748 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:29:36.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:29:36.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:36.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:37.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:38.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:38.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:38.250 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:29:38.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:29:38.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:38.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:38.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:39.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:39.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:39.831 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:29:39.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:29:39.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:39.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:40.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:41.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:41.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:41.442 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:29:41.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:29:41.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:41.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:41.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:42.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:42.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:42.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:29:42.892 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:29:42.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:42.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:43.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:44.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:44.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:44.449 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:29:44.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:29:44.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:44.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:44.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:45.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:45.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:45.891 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:29:45.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:29:45.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:45.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:46.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:47.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:47.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:47.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:29:47.508 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:29:47.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:47.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:48.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:49.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:49.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:49.326 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:29:49.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:29:49.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:49.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:50.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:51.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:51.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:51.107 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:29:51.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:29:51.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:51.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:51.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:52.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:52.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:52.995 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:29:52.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:29:52.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:52.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:53.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:54.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:54.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:54.913 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:29:54.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:29:54.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:54.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:56.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:57.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:57.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:57.065 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:29:57.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:29:57.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:57.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:57.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:29:58.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:29:58.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:29:58.791 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:29:58.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:29:58.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:29:58.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:29:59.249 INFO:tasks.workunit.client.0.smithi137.stdout:osd.5 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3535082814,v1:127.0.0.1:6843/3535082814] [v2:127.0.0.1:6844/3535082814,v1:127.0.0.1:6845/3535082814] exists,up 4b0156a0-f700-4565-8a03-decb441e93f5 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-12T16:29:59.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:29:59.255 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:29:59.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:29:59.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:29:59.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-04-12T16:29:59.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:29:59.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ad35e07-f818-49be-9bc0-85629ae0f6bb 2024-04-12T16:29:59.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 8ad35e07-f818-49be-9bc0-85629ae0f6bb' 2024-04-12T16:29:59.257 INFO:tasks.workunit.client.0.smithi137.stdout:add osd6 8ad35e07-f818-49be-9bc0-85629ae0f6bb 2024-04-12T16:29:59.257 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:29:59.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHYRlm7SaKEBAADDQ2cU5FLgCeC9q6arPDqg== 2024-04-12T16:29:59.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHYRlm7SaKEBAADDQ2cU5FLgCeC9q6arPDqg=="}' 2024-04-12T16:29:59.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ad35e07-f818-49be-9bc0-85629ae0f6bb -i td/test-erasure-eio/6/new.json 2024-04-12T16:29:59.918 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:29:59.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-04-12T16:29:59.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHYRlm7SaKEBAADDQ2cU5FLgCeC9q6arPDqg== --osd-uuid 8ad35e07-f818-49be-9bc0-85629ae0f6bb 2024-04-12T16:29:59.965 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:59.965+0000 7f28d8c187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:59.967 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:59.965+0000 7f28d8c187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:59.968 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:59.969+0000 7f28d8c187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:29:59.968 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:59.969+0000 7f28d8c187c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:29:59.968 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:59.969+0000 7f28d8c187c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:29:59.969 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:29:59.969+0000 7f28d8c187c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:30:00.152 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:00.153+0000 7f28d8c187c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-04-12T16:30:07.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-04-12T16:30:07.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:30:07.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-12T16:30:07.345 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd6 key to auth repository 2024-04-12T16:30:07.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:30:07.918 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.6 2024-04-12T16:30:07.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-12T16:30:07.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:30:07.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:30:07.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:30:07.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:30:07.950 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:07.949+0000 7fca16d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:30:07.974 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:07.973+0000 7fca16d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:30:07.976 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:07.977+0000 7fca16d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:30:08.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-12T16:30:08.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:30:08.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-12T16:30:08.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:30:08.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:30:08.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:08.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:30:08.322 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:30:08.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:08.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:08.530 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:08.529+0000 7fca16d0a7c0 -1 Falling back to public interface 2024-04-12T16:30:08.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:09.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:09.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:09.780 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:30:09.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:30:09.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:09.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:10.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:11.127 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:11.125+0000 7fca16d0a7c0 -1 osd.6 0 log_to_monitors true 2024-04-12T16:30:11.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:11.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:11.202 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:30:11.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:30:11.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:11.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:11.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:12.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:12.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:12.930 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:30:12.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:30:12.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:12.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:13.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:14.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:14.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:14.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:30:14.725 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:30:14.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:14.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:15.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:16.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:16.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:16.311 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:30:16.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:30:16.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:16.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:16.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:17.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:17.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:17.821 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:30:17.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:30:17.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:17.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:18.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:19.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:19.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:19.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:30:19.672 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:30:19.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:19.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:20.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:21.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:21.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:21.305 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:30:21.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:30:21.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:21.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:21.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:22.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:22.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:22.758 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:30:22.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:30:22.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:22.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:23.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:24.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:24.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:24.251 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:30:24.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:30:24.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:24.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:24.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:25.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:25.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:25.774 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:30:25.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:30:25.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:25.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:26.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:27.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:27.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:27.826 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:30:27.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:30:27.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:27.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:28.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:29.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:29.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:29.745 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:30:29.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:30:29.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:29.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:30.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:31.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:31.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:31.647 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:30:31.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:30:31.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:31.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:32.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:33.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:33.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:33.512 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:30:33.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:30:33.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:33.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:34.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:35.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:35.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:35.550 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:30:35.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:30:35.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:35.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:36.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:37.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:37.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:37.121 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:30:37.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:30:37.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:37.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:37.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:38.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:38.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:38.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:30:38.665 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:30:38.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:38.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:30:39.177 INFO:tasks.workunit.client.0.smithi137.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2797090196,v1:127.0.0.1:6851/2797090196] [v2:127.0.0.1:6852/2797090196,v1:127.0.0.1:6853/2797090196] exists,up 8ad35e07-f818-49be-9bc0-85629ae0f6bb 2024-04-12T16:30:39.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:30:39.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:30:39.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:30:39.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T16:30:39.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:30:39.178 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:30:39.179 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:30:39.179 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:30:39.179 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:30:39.179 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T16:30:39.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T16:30:39.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T16:30:39.301 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:30:39.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T16:30:39.321 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:26:15.025+0000 7fc2a9b967c0 0 load: jerasure load: lrc 2024-04-12T16:30:39.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-04-12T16:30:39.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-04-12T16:30:39.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:30:39.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T16:30:39.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-04-12T16:30:39.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T16:30:39.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-04-12T16:30:39.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T16:30:39.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-04-12T16:30:40.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T16:30:40.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T16:30:41.262 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T16:30:41.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:30:42.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T16:30:42.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:30:42.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:30:42.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:30:42.281 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:30:42.282 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:30:42.283 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:30:42.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:30:42.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:30:42.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:30:42.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:30:42.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:30:42.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:30:42.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:30:42.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:30:42.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:30:42.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:42.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:30:43.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803830 2024-04-12T16:30:43.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803830 2024-04-12T16:30:43.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830' 2024-04-12T16:30:43.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:43.122 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:30:43.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574894 2024-04-12T16:30:43.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574894 2024-04-12T16:30:43.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574894' 2024-04-12T16:30:43.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:43.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:30:43.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345962 2024-04-12T16:30:43.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345962 2024-04-12T16:30:43.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574894 2-85899345962' 2024-04-12T16:30:43.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:43.453 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:30:43.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018912 2024-04-12T16:30:43.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018912 2024-04-12T16:30:43.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574894 2-85899345962 3-128849018912' 2024-04-12T16:30:43.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:43.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:30:43.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691864 2024-04-12T16:30:43.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691864 2024-04-12T16:30:43.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574894 2-85899345962 3-128849018912 4-171798691864' 2024-04-12T16:30:43.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:43.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:30:43.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364818 2024-04-12T16:30:43.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364818 2024-04-12T16:30:43.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574894 2-85899345962 3-128849018912 4-171798691864 5-214748364818' 2024-04-12T16:30:43.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:43.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:30:44.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005062 2024-04-12T16:30:44.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005062 2024-04-12T16:30:44.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574894 2-85899345962 3-128849018912 4-171798691864 5-214748364818 6-261993005062' 2024-04-12T16:30:44.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:44.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803830 2024-04-12T16:30:44.069 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:44.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:30:44.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803830 2024-04-12T16:30:44.070 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:44.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803830 2024-04-12T16:30:44.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-04-12T16:30:44.072 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803830 2024-04-12T16:30:44.072 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:30:44.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803831 -lt 25769803830 2024-04-12T16:30:44.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:44.488 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574894 2024-04-12T16:30:44.488 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:44.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:30:44.490 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574894 2024-04-12T16:30:44.490 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:44.492 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574894 2024-04-12T16:30:44.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574894 2024-04-12T16:30:44.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574894' 2024-04-12T16:30:44.492 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:30:44.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574894 -lt 55834574894 2024-04-12T16:30:44.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:44.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345962 2024-04-12T16:30:44.907 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:44.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:30:44.909 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345962 2024-04-12T16:30:44.909 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:44.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345962 2024-04-12T16:30:44.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345962' 2024-04-12T16:30:44.912 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 85899345962 2024-04-12T16:30:44.912 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:30:45.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345962 -lt 85899345962 2024-04-12T16:30:45.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:45.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018912 2024-04-12T16:30:45.370 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:45.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:30:45.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018912 2024-04-12T16:30:45.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:45.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018912 2024-04-12T16:30:45.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018912' 2024-04-12T16:30:45.374 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 128849018912 2024-04-12T16:30:45.374 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:30:45.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018912 -lt 128849018912 2024-04-12T16:30:45.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:45.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-171798691864 2024-04-12T16:30:45.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:45.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:30:45.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-171798691864 2024-04-12T16:30:45.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:45.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691864 2024-04-12T16:30:45.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 171798691864' 2024-04-12T16:30:45.883 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 171798691864 2024-04-12T16:30:45.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:30:46.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691864 2024-04-12T16:30:46.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:46.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-214748364818 2024-04-12T16:30:46.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:46.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:30:46.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-214748364818 2024-04-12T16:30:46.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:46.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364818 2024-04-12T16:30:46.285 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 214748364818 2024-04-12T16:30:46.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 214748364818' 2024-04-12T16:30:46.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:30:46.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364819 -lt 214748364818 2024-04-12T16:30:46.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:46.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005062 2024-04-12T16:30:46.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:46.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:30:46.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005062 2024-04-12T16:30:46.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:46.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005062 2024-04-12T16:30:46.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005062' 2024-04-12T16:30:46.698 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 261993005062 2024-04-12T16:30:46.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:30:47.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005062 -lt 261993005062 2024-04-12T16:30:47.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:30:47.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:30:47.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:30:47.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:30:47.669 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:30:47.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:30:47.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:30:47.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:30:47.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:30:47.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:30:47.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:30:48.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:30:48.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:30:48.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:30:48.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:30:48.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:30:48.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:30:48.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:30:48.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-04-12T16:30:48.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-04-12T16:30:48.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-04-12T16:30:48.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:30:48.625 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-12T16:30:49.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: local primary_osd=3 2024-04-12T16:30:49.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-04-12T16:30:49.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:30:49.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:30:49.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:30:49.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:30:49.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:30:49.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:30:49.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:30:49.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-04-12T16:30:50.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:375: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-04-12T16:30:50.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:30:50.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-04-12T16:30:50.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:30:50.961 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T16:30:50.961 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:30:50.961 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:30:51.388 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2147483647 2024-04-12T16:30:51.388 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:30:51.388 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:30:51.388 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:30:51.388 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:30:51.388 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2147483647 5 1 0 6 2024-04-12T16:30:51.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2024-04-12T16:30:51.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:30:51.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=5 2024-04-12T16:30:51.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:30:51.389 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/5/type 2024-04-12T16:30:51.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:30:51.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-04-12T16:30:51.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:30:51.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=5 2024-04-12T16:30:51.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:30:51.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:30:51.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:30:51.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.5 2024-04-12T16:30:51.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-04-12T16:30:51.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-04-12T16:30:51.391 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:30:51.392 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:30:51.392 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:30:51.392 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.5.asok 2024-04-12T16:30:51.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:30:51.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:30:51.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:30:51.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:30:51.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.5 2024-04-12T16:30:51.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-04-12T16:30:51.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-04-12T16:30:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:30:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:30:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:30:51.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.5.asok 2024-04-12T16:30:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:30:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-04-12T16:30:51.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:378: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-04-12T16:30:51.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T16:30:51.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T16:30:51.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-12T16:30:51.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T16:30:51.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T16:30:51.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:30:51.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:30:51.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:30:51.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:30:51.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T16:30:51.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T16:30:51.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T16:30:51.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T16:30:51.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:30:51.639 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T16:30:51.639 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:30:51.639 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:30:51.639 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T16:30:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T16:30:51.641 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:30:51.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-12T16:30:51.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:30:51.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-04-12T16:30:51.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-12T16:30:51.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T16:30:51.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T16:30:51.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T16:30:51.670 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:51.669+0000 7f6aeb5227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:30:51.673 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:51.673+0000 7f6aeb5227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:30:51.678 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:51.673+0000 7f6aeb5227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:30:52.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-12T16:30:52.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:30:52.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:30:52.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:30:52.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:30:52.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:52.059 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:30:52.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:30:52.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:52.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:30:52.250 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:52.249+0000 7f6aeb5227c0 -1 Falling back to public interface 2024-04-12T16:30:52.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:53.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:53.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:53.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:30:53.611 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:30:53.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:53.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:30:54.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:55.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:55.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:55.018 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:30:55.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:30:55.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:55.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:30:55.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:30:55.427 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:30:55.425+0000 7f6aeb5227c0 -1 osd.3 69 log_to_monitors true 2024-04-12T16:30:56.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:30:56.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:30:56.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:30:56.429 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:30:56.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:30:56.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:30:56.859 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [30,69) [v2:127.0.0.1:6826/2898072456,v1:127.0.0.1:6827/2898072456] [v2:127.0.0.1:6828/2898072456,v1:127.0.0.1:6829/2898072456] exists,up 21e95c1f-4e62-4b9a-a28b-b6a24423750b 2024-04-12T16:30:56.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:30:56.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:30:56.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:30:56.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:381: TEST_ec_object_attr_read_error: wait_for_clean 2024-04-12T16:30:56.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:30:56.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:30:56.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:30:56.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:30:56.862 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:30:56.862 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:30:56.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:30:56.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:30:56.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:30:56.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:30:56.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:30:56.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:30:56.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:30:56.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:30:56.962 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:30:57.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:30:57.386 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:30:57.386 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:30:57.386 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:30:57.386 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:30:57.386 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:30:57.387 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:30:57.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:30:57.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:57.387 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:30:57.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803835 2024-04-12T16:30:57.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803835 2024-04-12T16:30:57.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835' 2024-04-12T16:30:57.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:57.648 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:30:57.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574899 2024-04-12T16:30:57.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574899 2024-04-12T16:30:57.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574899' 2024-04-12T16:30:57.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:57.803 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:30:57.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345967 2024-04-12T16:30:57.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345967 2024-04-12T16:30:57.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574899 2-85899345967' 2024-04-12T16:30:57.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:57.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:30:58.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612611 2024-04-12T16:30:58.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612611 2024-04-12T16:30:58.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574899 2-85899345967 3-313532612611' 2024-04-12T16:30:58.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:58.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:30:58.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691869 2024-04-12T16:30:58.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691869 2024-04-12T16:30:58.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574899 2-85899345967 3-313532612611 4-171798691869' 2024-04-12T16:30:58.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:58.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:30:58.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364823 2024-04-12T16:30:58.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364823 2024-04-12T16:30:58.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574899 2-85899345967 3-313532612611 4-171798691869 5-214748364823' 2024-04-12T16:30:58.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:30:58.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:30:58.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005066 2024-04-12T16:30:58.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005066 2024-04-12T16:30:58.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574899 2-85899345967 3-313532612611 4-171798691869 5-214748364823 6-261993005066' 2024-04-12T16:30:58.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:58.582 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:58.582 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803835 2024-04-12T16:30:58.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:30:58.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803835 2024-04-12T16:30:58.583 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:58.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803835 2024-04-12T16:30:58.585 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803835 2024-04-12T16:30:58.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-04-12T16:30:58.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:30:59.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803835 -lt 25769803835 2024-04-12T16:30:59.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:59.056 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574899 2024-04-12T16:30:59.056 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:59.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:30:59.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574899 2024-04-12T16:30:59.059 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:59.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574899 2024-04-12T16:30:59.061 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574899 2024-04-12T16:30:59.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574899' 2024-04-12T16:30:59.061 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:30:59.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574899 -lt 55834574899 2024-04-12T16:30:59.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:30:59.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345967 2024-04-12T16:30:59.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:30:59.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:30:59.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345967 2024-04-12T16:30:59.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:30:59.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345967 2024-04-12T16:30:59.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345967' 2024-04-12T16:30:59.474 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 85899345967 2024-04-12T16:30:59.475 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:30:59.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345965 -lt 85899345967 2024-04-12T16:30:59.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T16:31:00.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T16:31:00.900 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:31:01.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345968 -lt 85899345967 2024-04-12T16:31:01.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:31:01.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612611 2024-04-12T16:31:01.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:31:01.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:31:01.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612611 2024-04-12T16:31:01.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:31:01.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612611 2024-04-12T16:31:01.318 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 313532612611 2024-04-12T16:31:01.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612611' 2024-04-12T16:31:01.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:31:01.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612611 2024-04-12T16:31:01.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:31:01.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-171798691869 2024-04-12T16:31:01.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:31:01.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:31:01.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-171798691869 2024-04-12T16:31:01.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:31:01.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691869 2024-04-12T16:31:01.763 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 171798691869 2024-04-12T16:31:01.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 171798691869' 2024-04-12T16:31:01.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:31:02.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691870 -lt 171798691869 2024-04-12T16:31:02.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:31:02.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-214748364823 2024-04-12T16:31:02.192 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:31:02.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:31:02.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:31:02.195 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-214748364823 2024-04-12T16:31:02.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364823 2024-04-12T16:31:02.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 214748364823' 2024-04-12T16:31:02.196 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 214748364823 2024-04-12T16:31:02.196 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:31:02.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364824 -lt 214748364823 2024-04-12T16:31:02.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:31:02.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:31:02.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005066 2024-04-12T16:31:02.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:31:02.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005066 2024-04-12T16:31:02.616 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:31:02.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005066 2024-04-12T16:31:02.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005066' 2024-04-12T16:31:02.617 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 261993005066 2024-04-12T16:31:02.617 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:31:03.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005067 -lt 261993005066 2024-04-12T16:31:03.056 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:31:03.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:31:03.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:31:03.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:31:03.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:31:03.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:31:03.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:31:03.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:31:03.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:31:03.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:31:03.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:31:04.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:31:04.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:31:04.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:31:04.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:31:04.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:383: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T16:31:04.582 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-04-12T16:31:04.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T16:31:04.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T16:31:04.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-04-12T16:31:04.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:31:04.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T16:31:05.485 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T16:31:05.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-04-12T16:31:06.138 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile myprofile does not exist 2024-04-12T16:31:06.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-04-12T16:31:06.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:31:06.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:31:06.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:31:06.156 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:31:06.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:31:06.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:31:06.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:31:06.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:31:06.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:31:06.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:31:06.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:31:06.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:31:06.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:31:06.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:31:06.475 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:31:06.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:31:06.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:31:06.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:31:06.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:31:06.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:31:06.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:31:06.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:31:06.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:31:06.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:06.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:06.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:31:06.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:31:06.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:31:06.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T16:31:06.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T16:31:06.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:31:06.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:31:06.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:31:06.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:31:06.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:31:06.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:31:06.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:31:06.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:31:06.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:31:06.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:31:06.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:31:06.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:31:06.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:31:06.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:31:06.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:31:06.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:31:06.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:31:06.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:06.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:06.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:31:06.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:31:06.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:31:06.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T16:31:06.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:31:06.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:06.536 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:06.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T16:31:06.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:31:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:31:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:31:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T16:31:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T16:31:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T16:31:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:31:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:31:06.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:31:06.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T16:31:06.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T16:31:07.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:31:07.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:31:07.074 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:31:07.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:31:07.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:07.075 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:07.075 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:31:07.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:31:07.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:31:07.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:31:07.329 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:31:07.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:31:07.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:31:07.330 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:31:07.330 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:31:07.331 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:31:07.331 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:31:07.331 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:31:07.331 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:07.332 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:07.332 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:31:07.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:31:07.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T16:31:07.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:31:07.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:31:07.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:31:07.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:31:07.449 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:31:07.450 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:31:07.451 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:31:07.451 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:31:07.451 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:31:07.451 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:07.451 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:07.451 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:31:07.451 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:31:07.451 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T16:31:07.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T16:31:07.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T16:31:07.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:31:07.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:31:07.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:31:07.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T16:31:07.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:31:08.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:31:08.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:31:08.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:31:08.534 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:31:08.534 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:08.534 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:08.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:31:08.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:31:08.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:31:08.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T16:31:08.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:31:08.955 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:31:08.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:31:09.975 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T16:31:09.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:31:09.976 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:31:09.976 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:31:09.976 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:09.976 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:09.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:31:09.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T16:31:09.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T16:31:10.077 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:31:10.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T16:31:10.092 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:31:07.121+0000 7fe7b623ec80 0 load: jerasure load: lrc 2024-04-12T16:31:10.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-04-12T16:31:10.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-04-12T16:31:10.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-04-12T16:31:10.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-04-12T16:31:10.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-04-12T16:31:10.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T16:31:10.094 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-04-12T16:31:10.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-04-12T16:31:10.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:31:10.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T16:31:10.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:31:10.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:31:10.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:31:10.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:31:10.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:31:10.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:31:10.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:31:10.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:31:10.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T16:31:10.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:31:10.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9fdc964d-d7c6-423e-b2e6-4e5389874291 2024-04-12T16:31:10.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9fdc964d-d7c6-423e-b2e6-4e5389874291' 2024-04-12T16:31:10.102 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 9fdc964d-d7c6-423e-b2e6-4e5389874291 2024-04-12T16:31:10.102 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:31:10.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOYRlmBBtkBxAAUewwZL0O2xfHrt/XNdPLhA== 2024-04-12T16:31:10.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOYRlmBBtkBxAAUewwZL0O2xfHrt/XNdPLhA=="}' 2024-04-12T16:31:10.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9fdc964d-d7c6-423e-b2e6-4e5389874291 -i td/test-erasure-eio/0/new.json 2024-04-12T16:31:10.481 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:31:10.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T16:31:10.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOYRlmBBtkBxAAUewwZL0O2xfHrt/XNdPLhA== --osd-uuid 9fdc964d-d7c6-423e-b2e6-4e5389874291 2024-04-12T16:31:10.533 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:10.533+0000 7fa3d70e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:10.566 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:10.565+0000 7fa3d70e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:10.569+0000 7fa3d70e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:10.569+0000 7fa3d70e57c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:31:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:10.569+0000 7fa3d70e57c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:31:10.569 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:10.569+0000 7fa3d70e57c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:31:10.606 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:10.605+0000 7fa3d70e57c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T16:31:18.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T16:31:18.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:31:18.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:31:18.446 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:31:18.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:31:19.016 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:31:19.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:31:19.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:31:19.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:31:19.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:31:19.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:31:19.043 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:19.041+0000 7f3a7c2bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:19.045 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:19.045+0000 7f3a7c2bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:19.046 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:19.045+0000 7f3a7c2bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:19.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:31:19.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:31:19.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:31:19.415 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:31:19.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:31:19.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:31:19.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:19.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:31:19.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:19.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:19.601 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:19.601+0000 7f3a7c2bb7c0 -1 Falling back to public interface 2024-04-12T16:31:19.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:20.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:20.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:20.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:31:20.820 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:31:20.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:20.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:21.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:22.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:22.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:22.225 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:31:22.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:31:22.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:22.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:22.412 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:22.413+0000 7f3a7c2bb7c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:31:22.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:23.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:23.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:23.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:31:23.664 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:31:23.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:23.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:24.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:25.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:25.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:25.111 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:31:25.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:31:25.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:25.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:25.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:26.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:26.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:26.690 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:31:26.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:31:26.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:26.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:27.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:28.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:28.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:28.171 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:31:28.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:31:28.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:28.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:28.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:29.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:29.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:29.841 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:31:29.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:31:29.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:29.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:30.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:31.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:31.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:31.496 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:31:31.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:31:31.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:31.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:31.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:32.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:32.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:32.917 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:31:32.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:31:32.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:32.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:33.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:34.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:34.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:34.609 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:31:34.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:31:34.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:34.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:35.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:36.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:36.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:36.034 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:31:36.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:31:36.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:36.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:36.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:37.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:37.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:37.480 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:31:37.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:31:37.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:37.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:38.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:39.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:39.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:39.124 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:31:39.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:31:39.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:39.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:39.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:40.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:40.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:40.677 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:31:40.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:31:40.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:40.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:41.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:42.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:42.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:42.553 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:31:42.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:31:42.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:42.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:43.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:44.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:44.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:44.351 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:31:44.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:31:44.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:44.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:45.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:46.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:46.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:46.184 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:31:46.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:31:46.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:46.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:47.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:48.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:48.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:48.101 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:31:48.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:31:48.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:48.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:48.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:49.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:49.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:49.992 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:31:49.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:31:49.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:49.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:51.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:52.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:52.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:52.172 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:31:52.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:31:52.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:52.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:52.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:31:53.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:31:53.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:31:53.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:31:53.811 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:31:53.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:31:53.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:31:54.511 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3291886220,v1:127.0.0.1:6803/3291886220] [v2:127.0.0.1:6804/3291886220,v1:127.0.0.1:6805/3291886220] exists,up 9fdc964d-d7c6-423e-b2e6-4e5389874291 2024-04-12T16:31:54.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:31:54.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:31:54.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:31:54.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T16:31:54.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:31:54.515 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:31:54.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=786521f0-9407-4055-ac2e-0fec8f8b05e6 2024-04-12T16:31:54.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 786521f0-9407-4055-ac2e-0fec8f8b05e6' 2024-04-12T16:31:54.518 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 786521f0-9407-4055-ac2e-0fec8f8b05e6 2024-04-12T16:31:54.518 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:31:54.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6YRlm6WBeIBAAshFcXLfDALuJq0JBiTfc2g== 2024-04-12T16:31:54.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6YRlm6WBeIBAAshFcXLfDALuJq0JBiTfc2g=="}' 2024-04-12T16:31:54.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 786521f0-9407-4055-ac2e-0fec8f8b05e6 -i td/test-erasure-eio/1/new.json 2024-04-12T16:31:55.186 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:31:55.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T16:31:55.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6YRlm6WBeIBAAshFcXLfDALuJq0JBiTfc2g== --osd-uuid 786521f0-9407-4055-ac2e-0fec8f8b05e6 2024-04-12T16:31:55.238 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:55.237+0000 7fa6e32db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:55.275 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:55.277+0000 7fa6e32db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:55.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:55.277+0000 7fa6e32db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:31:55.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:55.277+0000 7fa6e32db7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:31:55.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:55.277+0000 7fa6e32db7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:31:55.277 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:55.277+0000 7fa6e32db7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:31:55.384 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:31:55.385+0000 7fa6e32db7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T16:32:03.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T16:32:03.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:32:03.121 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:32:03.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:32:03.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:32:03.700 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:32:03.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:32:03.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:32:03.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:32:03.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:32:03.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:32:03.731 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:03.733+0000 7f7ebbabe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:03.734 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:03.733+0000 7f7ebbabe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:03.735 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:03.737+0000 7f7ebbabe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:04.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:32:04.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:32:04.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:32:04.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:32:04.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:32:04.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:04.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:32:04.124 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:32:04.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:04.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:04.293 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:04.293+0000 7f7ebbabe7c0 -1 Falling back to public interface 2024-04-12T16:32:04.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:05.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:05.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:05.572 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:32:05.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:32:05.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:05.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:05.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:06.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:06.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:06.985 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:32:06.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:32:06.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:06.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:07.015 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:07.013+0000 7f7ebbabe7c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:32:07.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:08.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:08.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:08.450 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:32:08.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:32:08.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:08.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:08.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:09.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:09.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:09.912 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:32:09.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:32:09.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:09.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:10.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:11.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:11.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:11.587 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:32:11.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:32:11.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:11.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:12.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:13.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:13.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:13.186 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:32:13.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:32:13.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:13.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:13.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:14.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:14.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:14.825 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:32:14.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:32:14.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:14.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:15.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:16.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:16.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:16.379 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:32:16.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:32:16.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:16.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:16.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:17.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:17.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:17.896 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:32:17.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:32:17.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:17.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:18.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:19.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:19.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:19.501 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:32:19.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:32:19.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:19.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:20.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:21.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:21.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:21.052 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:32:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:32:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:21.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:22.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:22.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:22.589 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:32:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:32:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:23.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:24.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:24.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:24.155 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:32:24.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:32:24.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:24.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:24.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:25.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:25.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:25.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:32:25.745 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:32:25.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:25.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:26.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:27.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:27.313 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:27.313 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:32:27.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:32:27.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:27.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:28.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:29.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:29.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:29.133 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:32:29.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:32:29.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:29.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:29.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:30.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:30.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:30.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:32:30.966 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:32:30.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:30.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:31.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:32.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:32.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:32.980 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:32:32.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:32:32.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:32.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:33.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:34.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:34.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:34.891 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:32:34.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:32:34.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:34.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:35.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:36.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:36.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:36.891 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:32:36.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:32:36.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:36.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:38.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:39.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:39.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:39.038 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:32:39.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:32:39.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:39.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:32:39.734 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4169342878,v1:127.0.0.1:6811/4169342878] [v2:127.0.0.1:6812/4169342878,v1:127.0.0.1:6813/4169342878] exists,up 786521f0-9407-4055-ac2e-0fec8f8b05e6 2024-04-12T16:32:39.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:32:39.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:32:39.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:32:39.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:32:39.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:32:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:32:39.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:32:39.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f758a46e-3956-4550-9338-e30ead3c3964 2024-04-12T16:32:39.742 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 f758a46e-3956-4550-9338-e30ead3c3964 2024-04-12T16:32:39.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f758a46e-3956-4550-9338-e30ead3c3964' 2024-04-12T16:32:39.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:32:39.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnYhlmS2y6LRAA7BcVA4TjB1q89VrY3sJF7w== 2024-04-12T16:32:39.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnYhlmS2y6LRAA7BcVA4TjB1q89VrY3sJF7w=="}' 2024-04-12T16:32:39.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f758a46e-3956-4550-9338-e30ead3c3964 -i td/test-erasure-eio/2/new.json 2024-04-12T16:32:40.905 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:32:40.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T16:32:40.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnYhlmS2y6LRAA7BcVA4TjB1q89VrY3sJF7w== --osd-uuid f758a46e-3956-4550-9338-e30ead3c3964 2024-04-12T16:32:40.959 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:40.957+0000 7fb76edfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:41.009+0000 7fb76edfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:41.012 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:41.013+0000 7fb76edfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:41.012 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:41.013+0000 7fb76edfe7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:32:41.012 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:41.013+0000 7fb76edfe7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:32:41.013 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:41.013+0000 7fb76edfe7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:32:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:41.097+0000 7fb76edfe7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T16:32:49.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T16:32:49.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:32:49.278 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:32:49.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:32:49.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:32:49.858 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:32:49.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:32:49.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:32:49.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:32:49.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:32:49.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:32:49.890 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:49.889+0000 7f4d571867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:49.919 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:49.921+0000 7f4d571867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:49.920 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:49.921+0000 7f4d571867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:32:50.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:32:50.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:32:50.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:32:50.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:32:50.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:32:50.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:50.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:32:50.282 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:32:50.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:50.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:32:50.453 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:50.453+0000 7f4d571867c0 -1 Falling back to public interface 2024-04-12T16:32:50.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:51.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:51.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:51.754 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:32:51.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:32:51.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:51.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:32:52.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:53.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:53.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:53.181 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:32:53.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:32:53.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:53.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:32:53.258 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:32:53.257+0000 7f4d571867c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:32:53.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:54.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:54.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:54.723 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:32:54.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:32:54.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:54.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:32:55.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:56.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:56.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:56.234 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:32:56.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:32:56.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:56.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:32:56.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:57.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:57.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:57.959 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:32:57.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:32:57.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:57.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:32:58.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:32:59.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:32:59.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:32:59.559 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:32:59.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:32:59.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:32:59.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:00.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:01.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:01.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:01.129 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:33:01.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:33:01.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:01.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:01.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:02.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:02.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:02.900 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:33:02.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:33:02.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:02.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:03.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:04.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:04.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:04.630 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:33:04.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:33:04.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:04.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:05.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:06.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:06.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:06.070 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:33:06.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:33:06.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:06.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:06.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:07.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:07.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:07.764 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:33:07.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:33:07.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:07.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:08.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:09.299 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:33:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:33:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:10.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:11.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:11.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:11.111 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:33:11.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:33:11.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:11.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:11.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:12.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:12.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:12.766 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:33:12.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:33:12.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:12.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:13.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:14.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:14.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:14.876 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:33:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:33:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:15.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:16.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:16.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:16.695 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:33:16.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:33:16.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:16.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:17.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:18.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:18.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:18.754 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:33:18.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:33:18.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:18.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:19.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:20.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:20.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:20.827 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:33:20.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:33:20.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:20.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:21.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:22.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:22.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:22.827 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:33:22.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:33:22.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:22.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:23.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:24.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:24.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:24.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:33:24.659 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:33:24.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:24.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:25.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:26.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:26.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:26.221 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:33:26.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:33:26.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:26.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:33:26.831 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3056505425,v1:127.0.0.1:6819/3056505425] [v2:127.0.0.1:6820/3056505425,v1:127.0.0.1:6821/3056505425] exists,up f758a46e-3956-4550-9338-e30ead3c3964 2024-04-12T16:33:26.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:33:26.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T16:33:26.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:33:26.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:33:26.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T16:33:26.838 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:33:26.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c4f7417-1b8d-4073-960d-9965cdb64293 2024-04-12T16:33:26.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 7c4f7417-1b8d-4073-960d-9965cdb64293' 2024-04-12T16:33:26.838 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 7c4f7417-1b8d-4073-960d-9965cdb64293 2024-04-12T16:33:26.838 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:33:26.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWYhlmUfdnMxAAVwFjN2FRQ4dQJagWsHDb6A== 2024-04-12T16:33:26.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWYhlmUfdnMxAAVwFjN2FRQ4dQJagWsHDb6A=="}' 2024-04-12T16:33:26.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c4f7417-1b8d-4073-960d-9965cdb64293 -i td/test-erasure-eio/3/new.json 2024-04-12T16:33:27.611 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:33:27.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T16:33:27.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWYhlmUfdnMxAAVwFjN2FRQ4dQJagWsHDb6A== --osd-uuid 7c4f7417-1b8d-4073-960d-9965cdb64293 2024-04-12T16:33:27.663 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:27.666+0000 7f2a18e447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:33:27.665 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:27.666+0000 7f2a18e447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:33:27.666 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:27.666+0000 7f2a18e447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:33:27.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:27.670+0000 7f2a18e447c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:33:27.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:27.670+0000 7f2a18e447c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:33:27.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:27.670+0000 7f2a18e447c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:33:27.736 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:27.738+0000 7f2a18e447c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T16:33:35.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T16:33:35.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:33:35.410 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T16:33:35.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T16:33:35.410 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:33:35.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T16:33:35.979 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:33:35.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:33:35.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:33:35.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:33:35.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:33:36.010 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:36.010+0000 7ff6c00497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:33:36.012 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:36.014+0000 7ff6c00497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:33:36.013 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:36.014+0000 7ff6c00497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:33:36.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T16:33:36.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:33:36.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:33:36.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:33:36.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:33:36.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:36.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:33:36.392 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:33:36.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:36.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:36.572 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:36.574+0000 7ff6c00497c0 -1 Falling back to public interface 2024-04-12T16:33:36.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:37.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:37.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:37.829 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:33:37.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:33:37.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:37.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:38.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:39.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:39.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:39.253 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:33:39.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:33:39.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:39.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:39.512 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:33:39.514+0000 7ff6c00497c0 -1 osd.3 0 log_to_monitors true 2024-04-12T16:33:39.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:40.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:40.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:40.756 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:33:40.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:33:40.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:40.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:41.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:42.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:42.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:42.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:33:42.229 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:33:42.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:42.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:43.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:44.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:44.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:44.083 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:33:44.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:33:44.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:44.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:44.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:45.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:45.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:45.777 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:33:45.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:33:45.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:45.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:46.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:47.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:47.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:47.285 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:33:47.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:33:47.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:47.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:47.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:48.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:48.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:48.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:33:48.814 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:33:48.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:48.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:49.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:50.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:50.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:50.276 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:33:50.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:33:50.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:50.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:50.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:51.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:51.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:51.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:33:51.972 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:33:51.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:51.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:52.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:53.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:53.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:53.542 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:33:53.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:33:53.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:53.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:54.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:55.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:55.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:55.063 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:33:55.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:33:55.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:55.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:55.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:56.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:56.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:56.502 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:33:56.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:33:56.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:56.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:57.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:58.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:58.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:58.148 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:33:58.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:33:58.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:58.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:33:58.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:33:59.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:33:59.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:33:59.699 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:33:59.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:33:59.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:33:59.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:34:00.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:01.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:01.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:01.550 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:34:01.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:34:01.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:01.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:34:02.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:03.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:03.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:03.093 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:34:03.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:34:03.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:03.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:34:03.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:04.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:04.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:04.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:34:04.769 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:34:04.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:04.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:34:05.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:06.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:06.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:06.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:34:06.696 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:34:06.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:06.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:34:07.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:08.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:08.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:08.770 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:34:08.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:34:08.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:08.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:34:09.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:10.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:10.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:10.942 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:34:10.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:34:10.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:10.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:34:11.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:12.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:12.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:12.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:34:12.527 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:34:12.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:12.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:34:13.241 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1431011437,v1:127.0.0.1:6827/1431011437] [v2:127.0.0.1:6828/1431011437,v1:127.0.0.1:6829/1431011437] exists,up 7c4f7417-1b8d-4073-960d-9965cdb64293 2024-04-12T16:34:13.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:34:13.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:34:13.246 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:34:13.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:34:13.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:34:13.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:34:13.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:34:13.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:34:13.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-04-12T16:34:13.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:34:13.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4603b9d2-8992-4bd5-a8e5-75e5c968165e 2024-04-12T16:34:13.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 4603b9d2-8992-4bd5-a8e5-75e5c968165e' 2024-04-12T16:34:13.249 INFO:tasks.workunit.client.0.smithi137.stdout:add osd4 4603b9d2-8992-4bd5-a8e5-75e5c968165e 2024-04-12T16:34:13.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:34:13.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFYhlmfjpTEBAADSkZojs52cY3gxXlCxD3FQ== 2024-04-12T16:34:13.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFYhlmfjpTEBAADSkZojs52cY3gxXlCxD3FQ=="}' 2024-04-12T16:34:13.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4603b9d2-8992-4bd5-a8e5-75e5c968165e -i td/test-erasure-eio/4/new.json 2024-04-12T16:34:14.012 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:34:14.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-04-12T16:34:14.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFYhlmfjpTEBAADSkZojs52cY3gxXlCxD3FQ== --osd-uuid 4603b9d2-8992-4bd5-a8e5-75e5c968165e 2024-04-12T16:34:14.063 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:14.066+0000 7f6636acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:34:14.066 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:14.066+0000 7f6636acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:34:14.068 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:14.070+0000 7f6636acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:34:14.068 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:14.070+0000 7f6636acf7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:34:14.068 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:14.070+0000 7f6636acf7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:34:14.068 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:14.070+0000 7f6636acf7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:34:14.260 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:14.262+0000 7f6636acf7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-04-12T16:34:22.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-04-12T16:34:22.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:34:22.550 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd4 key to auth repository 2024-04-12T16:34:22.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-12T16:34:22.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:34:23.122 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.4 2024-04-12T16:34:23.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-12T16:34:23.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:34:23.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:34:23.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:34:23.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:34:23.154 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:23.154+0000 7f05697d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:34:23.156 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:23.158+0000 7f05697d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:34:23.157 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:23.158+0000 7f05697d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:34:23.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-12T16:34:23.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:34:23.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-12T16:34:23.543 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:34:23.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:34:23.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:34:23.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:23.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:34:23.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:23.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:23.696 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:23.698+0000 7f05697d87c0 -1 Falling back to public interface 2024-04-12T16:34:24.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:25.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:25.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:25.003 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:34:25.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:34:25.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:25.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:25.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:26.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:26.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:26.408 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:34:26.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:34:26.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:26.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:26.663 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:34:26.666+0000 7f05697d87c0 -1 osd.4 0 log_to_monitors true 2024-04-12T16:34:26.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:27.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:27.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:27.820 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:34:27.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:34:27.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:27.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:28.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:29.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:29.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:29.580 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:34:29.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:34:29.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:29.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:30.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:31.547 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:34:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:34:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:32.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:33.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:33.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:33.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:34:33.535 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:34:33.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:33.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:34.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:35.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:35.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:35.034 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:34:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:34:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:35.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:36.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:36.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:36.672 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:34:36.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:34:36.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:36.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:37.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:38.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:38.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:38.196 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:34:38.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:34:38.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:38.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:38.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:39.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:39.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:39.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:34:39.923 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:34:39.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:39.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:40.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:41.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:41.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:41.452 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:34:41.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:34:41.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:41.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:42.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:43.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:43.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:43.097 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:34:43.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:34:43.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:43.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:43.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:44.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:44.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:44.589 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:34:44.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:34:44.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:44.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:45.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:46.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:46.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:46.169 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:34:46.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:34:46.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:46.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:46.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:47.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:47.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:47.779 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:34:47.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:34:47.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:47.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:48.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:49.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:49.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:49.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:34:49.672 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:34:49.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:49.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:50.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:51.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:51.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:51.147 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:34:51.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:34:51.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:51.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:52.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:53.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:53.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:53.049 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:34:53.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:34:53.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:53.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:54.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:55.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:55.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:55.161 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:34:55.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:34:55.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:55.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:56.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:57.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:57.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:57.056 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:34:57.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:34:57.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:57.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:57.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:34:58.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:34:58.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:34:58.927 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:34:58.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:34:58.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:34:58.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:34:59.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:00.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:00.566 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:35:00.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:35:00.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:00.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:35:01.101 INFO:tasks.workunit.client.0.smithi137.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3992109904,v1:127.0.0.1:6835/3992109904] [v2:127.0.0.1:6836/3992109904,v1:127.0.0.1:6837/3992109904] exists,up 4603b9d2-8992-4bd5-a8e5-75e5c968165e 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:35:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:35:01.105 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:35:01.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:35:01.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:35:01.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:35:01.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:35:01.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:35:01.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc6b081e-6eaf-4f6e-8836-b844253fa1e2 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 bc6b081e-6eaf-4f6e-8836-b844253fa1e2' 2024-04-12T16:35:01.108 INFO:tasks.workunit.client.0.smithi137.stdout:add osd5 bc6b081e-6eaf-4f6e-8836-b844253fa1e2 2024-04-12T16:35:01.109 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:35:01.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1YhlmIV/6BxAAip+A3mYuF0FvBW5P4AA3Cg== 2024-04-12T16:35:01.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1YhlmIV/6BxAAip+A3mYuF0FvBW5P4AA3Cg=="}' 2024-04-12T16:35:01.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc6b081e-6eaf-4f6e-8836-b844253fa1e2 -i td/test-erasure-eio/5/new.json 2024-04-12T16:35:02.167 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:35:02.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-04-12T16:35:02.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1YhlmIV/6BxAAip+A3mYuF0FvBW5P4AA3Cg== --osd-uuid bc6b081e-6eaf-4f6e-8836-b844253fa1e2 2024-04-12T16:35:02.214 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:02.218+0000 7fa7861ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:02.216 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:02.218+0000 7fa7861ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:02.217 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:02.218+0000 7fa7861ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:02.218 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:02.222+0000 7fa7861ca7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:35:02.218 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:02.222+0000 7fa7861ca7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:35:02.218 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:02.222+0000 7fa7861ca7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:35:02.385 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:02.386+0000 7fa7861ca7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-04-12T16:35:09.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-04-12T16:35:09.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:35:09.816 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd5 key to auth repository 2024-04-12T16:35:09.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-12T16:35:09.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:35:10.395 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.5 2024-04-12T16:35:10.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-12T16:35:10.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:35:10.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:35:10.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:35:10.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:35:10.427 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:10.430+0000 7f103b73f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:10.429 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:10.430+0000 7f103b73f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:10.431 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:10.434+0000 7f103b73f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:10.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-12T16:35:10.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:35:10.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-12T16:35:10.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:35:10.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:35:10.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:10.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:35:10.803 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:35:10.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:10.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:10.979 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:10.982+0000 7f103b73f7c0 -1 Falling back to public interface 2024-04-12T16:35:11.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:12.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:12.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:12.228 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:35:12.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:35:12.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:12.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:12.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:13.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:13.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:13.655 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:35:13.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:35:13.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:13.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:13.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:13.670+0000 7f103b73f7c0 -1 osd.5 0 log_to_monitors true 2024-04-12T16:35:14.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:15.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:15.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:15.223 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:35:15.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:35:15.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:15.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:15.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:16.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:16.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:16.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:35:16.748 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:35:16.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:16.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:17.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:18.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:18.731 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:35:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:35:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:18.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:19.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:20.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:20.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:20.286 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:35:20.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:35:20.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:20.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:20.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:21.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:21.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:21.880 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:35:21.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:35:21.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:21.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:22.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:23.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:23.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:23.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:35:23.444 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:35:23.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:23.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:24.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:25.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:25.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:25.010 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:35:25.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:35:25.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:25.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:25.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:26.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:26.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:26.437 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:35:26.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:35:26.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:26.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:27.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:28.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:28.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:28.041 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:35:28.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:35:28.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:28.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:28.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:29.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:29.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:29.600 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:35:29.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:35:29.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:29.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:30.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:31.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:31.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:31.109 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:35:31.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:35:31.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:31.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:31.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:32.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:32.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:32.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:35:32.537 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:35:32.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:32.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:33.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:34.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:34.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:34.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:35:34.347 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:35:34.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:34.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:35.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:36.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:36.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:36.171 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:35:36.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:35:36.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:36.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:37.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:38.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:38.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:38.130 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:35:38.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:35:38.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:38.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:39.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:40.506 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:35:40.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:35:40.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:40.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:41.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:42.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:42.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:42.372 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:35:42.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:35:42.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:42.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:43.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:44.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:44.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:44.489 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:35:44.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:35:44.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:44.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:35:44.924 INFO:tasks.workunit.client.0.smithi137.stdout:osd.5 up in weight 1 up_from 47 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/950540094,v1:127.0.0.1:6843/950540094] [v2:127.0.0.1:6844/950540094,v1:127.0.0.1:6845/950540094] exists,up bc6b081e-6eaf-4f6e-8836-b844253fa1e2 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:35:44.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:35:44.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:35:44.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-04-12T16:35:44.931 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:35:44.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7fdb519-c276-4073-987f-503b6bae1c58 2024-04-12T16:35:44.931 INFO:tasks.workunit.client.0.smithi137.stdout:add osd6 e7fdb519-c276-4073-987f-503b6bae1c58 2024-04-12T16:35:44.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 e7fdb519-c276-4073-987f-503b6bae1c58' 2024-04-12T16:35:44.932 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:35:44.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDgYhlmUt0EORAAJYIfuEo3ChX1GqVoDtNdkg== 2024-04-12T16:35:44.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDgYhlmUt0EORAAJYIfuEo3ChX1GqVoDtNdkg=="}' 2024-04-12T16:35:44.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7fdb519-c276-4073-987f-503b6bae1c58 -i td/test-erasure-eio/6/new.json 2024-04-12T16:35:46.206 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:35:46.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-04-12T16:35:46.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgYhlmUt0EORAAJYIfuEo3ChX1GqVoDtNdkg== --osd-uuid e7fdb519-c276-4073-987f-503b6bae1c58 2024-04-12T16:35:46.262 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:46.266+0000 7f899199f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:46.293 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:46.294+0000 7f899199f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:46.295 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:46.298+0000 7f899199f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:46.296 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:46.298+0000 7f899199f7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:35:46.296 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:46.298+0000 7f899199f7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:35:46.296 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:46.298+0000 7f899199f7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:35:46.477 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:46.482+0000 7f899199f7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-04-12T16:35:53.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-04-12T16:35:53.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:35:53.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-12T16:35:53.659 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd6 key to auth repository 2024-04-12T16:35:53.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:35:54.233 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.6 2024-04-12T16:35:54.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-12T16:35:54.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:35:54.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:35:54.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:35:54.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:35:54.265 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:54.266+0000 7f1b1d5b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:54.267 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:54.270+0000 7f1b1d5b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:54.268 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:54.270+0000 7f1b1d5b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:35:54.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-12T16:35:54.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:35:54.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-12T16:35:54.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:35:54.651 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:35:54.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:35:54.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:54.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:35:54.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:54.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:35:54.819 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:54.822+0000 7f1b1d5b87c0 -1 Falling back to public interface 2024-04-12T16:35:55.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:56.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:56.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:56.049 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:35:56.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:35:56.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:56.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:35:56.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:57.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:57.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:57.453 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:35:57.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:35:57.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:57.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:35:57.521 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:35:57.522+0000 7f1b1d5b87c0 -1 osd.6 0 log_to_monitors true 2024-04-12T16:35:57.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:35:58.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:35:58.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:35:58.956 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:35:58.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:35:58.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:35:58.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:35:59.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:00.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:00.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:00.504 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:36:00.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:36:00.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:00.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:00.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:01.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:01.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:01.996 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:36:01.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:36:01.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:01.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:02.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:03.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:03.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:03.709 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:36:03.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:36:03.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:03.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:04.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:05.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:05.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:05.215 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:36:05.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:36:05.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:05.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:05.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:06.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:06.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:06.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:36:06.730 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:36:06.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:06.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:07.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:08.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:08.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:08.203 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:36:08.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:36:08.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:08.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:08.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:09.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:09.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:09.790 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:36:09.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:36:09.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:09.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:10.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:11.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:11.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:11.289 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:36:11.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:36:11.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:11.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:11.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:12.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:12.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:12.919 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:36:12.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:36:12.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:12.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:13.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:14.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:14.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:14.367 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:36:14.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:36:14.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:14.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:15.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:16.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:16.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:16.137 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:36:16.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:36:16.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:16.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:17.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:18.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:18.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:18.370 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:36:18.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:36:18.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:18.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:19.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:20.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:20.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:20.082 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:36:20.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:36:20.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:20.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:21.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:22.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:22.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:22.162 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:36:22.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:36:22.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:22.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:23.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:24.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:24.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:24.321 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:36:24.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:36:24.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:24.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:25.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:26.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:26.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:26.177 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:36:26.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:36:26.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:26.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:26.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:27.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:27.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:27.779 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:36:27.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:36:27.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:27.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:36:28.456 INFO:tasks.workunit.client.0.smithi137.stdout:osd.6 up in weight 1 up_from 56 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1088514004,v1:127.0.0.1:6851/1088514004] [v2:127.0.0.1:6852/1088514004,v1:127.0.0.1:6853/1088514004] exists,up e7fdb519-c276-4073-987f-503b6bae1c58 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/7 2024-04-12T16:36:28.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:36:28.460 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:36:28.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:36:28.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/7 2024-04-12T16:36:28.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:36:28.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d92e3acd-d705-483b-b2cd-861d9e1a3243 2024-04-12T16:36:28.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 d92e3acd-d705-483b-b2cd-861d9e1a3243' 2024-04-12T16:36:28.463 INFO:tasks.workunit.client.0.smithi137.stdout:add osd7 d92e3acd-d705-483b-b2cd-861d9e1a3243 2024-04-12T16:36:28.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:36:28.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMYxlmHCk+HRAA3SvgP0nyJ807A2Ga+wNHiA== 2024-04-12T16:36:28.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMYxlmHCk+HRAA3SvgP0nyJ807A2Ga+wNHiA=="}' 2024-04-12T16:36:28.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d92e3acd-d705-483b-b2cd-861d9e1a3243 -i td/test-erasure-eio/7/new.json 2024-04-12T16:36:29.179 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:36:29.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/7/new.json 2024-04-12T16:36:29.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMYxlmHCk+HRAA3SvgP0nyJ807A2Ga+wNHiA== --osd-uuid d92e3acd-d705-483b-b2cd-861d9e1a3243 2024-04-12T16:36:29.236 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:29.238+0000 7f42c6ef77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:36:29.238 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:29.242+0000 7f42c6ef77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:36:29.239 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:29.242+0000 7f42c6ef77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:36:29.239 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:29.242+0000 7f42c6ef77c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-04-12T16:36:29.240 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:29.242+0000 7f42c6ef77c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-04-12T16:36:29.240 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:29.242+0000 7f42c6ef77c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-04-12T16:36:29.430 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:29.434+0000 7f42c6ef77c0 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-04-12T16:36:38.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-04-12T16:36:38.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:36:38.509 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd7 key to auth repository 2024-04-12T16:36:38.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-04-12T16:36:38.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:36:39.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-04-12T16:36:39.083 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.7 2024-04-12T16:36:39.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:36:39.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:36:39.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:36:39.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:36:39.115 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:39.118+0000 7f4ef13637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:36:39.118 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:39.122+0000 7f4ef13637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:36:39.119 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:39.122+0000 7f4ef13637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:36:39.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-04-12T16:36:39.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:36:39.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-04-12T16:36:39.496 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:36:39.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:36:39.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:36:39.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:39.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:36:39.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:39.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:39.663 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:39.666+0000 7f4ef13637c0 -1 Falling back to public interface 2024-04-12T16:36:39.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:40.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:40.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:40.929 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:36:40.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:36:40.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:40.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:41.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:42.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:42.356 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:42.357 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:36:42.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:36:42.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:42.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:42.592 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:36:42.594+0000 7f4ef13637c0 -1 osd.7 0 log_to_monitors true 2024-04-12T16:36:42.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:43.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:43.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:43.870 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:36:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:36:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:43.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:44.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:45.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:45.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:45.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:36:45.897 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:36:45.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:45.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:47.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:48.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:48.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:48.118 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:36:48.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:36:48.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:48.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:48.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:49.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:49.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:49.807 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:36:49.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:36:49.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:49.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:50.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:51.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:51.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:51.452 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:36:51.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:36:51.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:51.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:52.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:53.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:53.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:53.239 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:36:53.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:36:53.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:53.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:53.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:54.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:54.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:54.905 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:36:54.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:36:54.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:54.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:55.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:56.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:56.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:56.337 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:36:56.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:36:56.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:56.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:56.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:57.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:57.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:57.966 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:36:57.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:36:57.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:57.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:36:58.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:36:59.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:36:59.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:36:59.680 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:36:59.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:36:59.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:36:59.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:00.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:01.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:01.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:01.460 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:37:01.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:37:01.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:01.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:02.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:03.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:03.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:03.257 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:37:03.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:37:03.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:03.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:04.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:05.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:05.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:05.516 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:37:05.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:37:05.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:05.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:06.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:07.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:07.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:07.154 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:37:07.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:37:07.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:07.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:08.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:09.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:09.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:09.022 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:37:09.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:37:09.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:09.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:09.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:10.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:10.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:10.621 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:37:10.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:37:10.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:10.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:11.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:12.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:12.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:12.227 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:37:12.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:37:12.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:12.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:13.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:14.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:14.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:14.113 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:37:14.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:37:14.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:14.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:14.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:15.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:15.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:15.807 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:37:15.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:37:15.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:15.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:16.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:17.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:17.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:17.715 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:37:17.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:37:17.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:17.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-12T16:37:18.310 INFO:tasks.workunit.client.0.smithi137.stdout:osd.7 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1977895470,v1:127.0.0.1:6859/1977895470] [v2:127.0.0.1:6860/1977895470,v1:127.0.0.1:6861/1977895470] exists,up d92e3acd-d705-483b-b2cd-861d9e1a3243 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/8 2024-04-12T16:37:18.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:37:18.315 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:37:18.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/8 2024-04-12T16:37:18.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:37:18.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42868cdc-158e-4c3e-bcbc-c8f53f7d17c0 2024-04-12T16:37:18.317 INFO:tasks.workunit.client.0.smithi137.stdout:add osd8 42868cdc-158e-4c3e-bcbc-c8f53f7d17c0 2024-04-12T16:37:18.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 42868cdc-158e-4c3e-bcbc-c8f53f7d17c0' 2024-04-12T16:37:18.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:37:18.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+YxlmWK85FBAAKnr6aFVXOfkcL0bGdNng3A== 2024-04-12T16:37:18.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+YxlmWK85FBAAKnr6aFVXOfkcL0bGdNng3A=="}' 2024-04-12T16:37:18.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42868cdc-158e-4c3e-bcbc-c8f53f7d17c0 -i td/test-erasure-eio/8/new.json 2024-04-12T16:37:19.197 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:37:19.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/8/new.json 2024-04-12T16:37:19.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+YxlmWK85FBAAKnr6aFVXOfkcL0bGdNng3A== --osd-uuid 42868cdc-158e-4c3e-bcbc-c8f53f7d17c0 2024-04-12T16:37:19.245 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:19.246+0000 7fc4bc21e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:37:19.247 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:19.250+0000 7fc4bc21e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:37:19.248 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:19.250+0000 7fc4bc21e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:37:19.249 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:19.250+0000 7fc4bc21e7c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-04-12T16:37:19.249 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:19.250+0000 7fc4bc21e7c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-04-12T16:37:19.249 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:19.250+0000 7fc4bc21e7c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-04-12T16:37:19.385 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:19.386+0000 7fc4bc21e7c0 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-04-12T16:37:28.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-04-12T16:37:28.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:37:28.507 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd8 key to auth repository 2024-04-12T16:37:28.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-04-12T16:37:28.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:37:29.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-04-12T16:37:29.081 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.8 2024-04-12T16:37:29.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:37:29.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:37:29.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:37:29.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:37:29.113 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:29.118+0000 7f15a89697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:37:29.115 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:29.118+0000 7f15a89697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:37:29.117 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:29.118+0000 7f15a89697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:37:29.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-04-12T16:37:29.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:37:29.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-04-12T16:37:29.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:37:29.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:37:29.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:29.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:37:29.547 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:37:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:29.678 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:29.682+0000 7f15a89697c0 -1 Falling back to public interface 2024-04-12T16:37:30.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:31.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:31.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:31.007 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:37:31.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:37:31.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:31.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:31.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:32.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:32.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:32.419 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:37:32.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:37:32.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:32.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:32.514 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:37:32.518+0000 7f15a89697c0 -1 osd.8 0 log_to_monitors true 2024-04-12T16:37:32.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:33.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:33.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:33.950 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:37:33.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:37:33.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:33.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:34.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:35.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:35.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:35.751 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:37:35.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:37:35.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:35.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:36.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:37.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:37.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:37.861 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:37:37.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:37:37.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:37.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:38.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:39.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:39.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:39.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:37:39.713 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:37:39.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:39.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:40.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:41.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:41.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:41.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:37:41.223 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:37:41.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:41.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:41.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:42.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:42.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:42.839 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:37:42.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:37:42.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:42.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:43.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:44.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:44.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:44.305 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:37:44.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:37:44.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:44.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:44.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:45.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:45.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:45.921 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:37:45.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:37:45.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:45.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:46.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:47.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:47.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:47.823 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:37:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:37:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:48.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:49.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:49.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:49.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:37:49.520 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:37:49.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:49.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:50.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:51.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:51.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:51.221 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:37:51.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:37:51.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:51.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:52.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:53.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:53.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:53.052 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:37:53.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:37:53.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:53.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:53.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:54.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:54.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:54.655 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:37:54.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:37:54.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:54.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:55.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:56.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:56.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:56.432 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:37:56.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:37:56.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:56.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:57.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:37:58.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:37:58.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:37:58.266 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:37:58.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:37:58.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:37:58.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:37:59.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:00.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:00.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:00.141 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:38:00.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:38:00.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:00.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:38:01.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:02.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:02.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:02.659 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:38:02.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:38:02.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:02.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:38:03.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:04.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:04.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:04.672 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:38:04.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:38:04.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:04.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:38:05.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:06.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:06.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:06.648 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:38:06.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:38:06.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:06.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:38:06.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:07.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:07.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:07.906 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:38:07.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:38:07.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:07.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-12T16:38:08.546 INFO:tasks.workunit.client.0.smithi137.stdout:osd.8 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/403462697,v1:127.0.0.1:6867/403462697] [v2:127.0.0.1:6868/403462697,v1:127.0.0.1:6869/403462697] exists,up 42868cdc-158e-4c3e-bcbc-c8f53f7d17c0 2024-04-12T16:38:08.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:38:08.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:38:08.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:38:08.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T16:38:08.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:38:08.548 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:38:08.548 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:08.549 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:08.549 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:08.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T16:38:08.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T16:38:08.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T16:38:08.654 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:38:08.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T16:38:08.675 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:31:19.605+0000 7f3a7c2bb7c0 0 load: jerasure load: lrc 2024-04-12T16:38:08.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-04-12T16:38:08.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-04-12T16:38:08.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:38:08.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T16:38:08.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-04-12T16:38:08.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T16:38:08.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-04-12T16:38:08.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T16:38:08.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-04-12T16:38:10.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T16:38:10.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T16:38:12.280 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T16:38:12.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:38:13.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T16:38:13.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:38:13.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:38:13.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:38:13.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:38:13.305 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:38:13.305 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:38:13.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:38:13.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:38:13.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:38:13.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:38:13.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:38:13.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:38:13.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:38:13.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:38:13.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:38:14.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:38:14.055 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:38:14.055 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:38:14.055 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:38:14.055 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:38:14.056 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:38:14.056 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:38:14.056 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:38:14.056 INFO:tasks.workunit.client.0.smithi137.stderr:8' 2024-04-12T16:38:14.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:38:14.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:14.057 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:38:14.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803863 2024-04-12T16:38:14.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803863 2024-04-12T16:38:14.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863' 2024-04-12T16:38:14.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:14.376 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:38:14.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574919 2024-04-12T16:38:14.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574919 2024-04-12T16:38:14.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863 1-55834574919' 2024-04-12T16:38:14.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:14.532 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:38:14.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378689 2024-04-12T16:38:14.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378689 2024-04-12T16:38:14.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863 1-55834574919 2-81604378689' 2024-04-12T16:38:14.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:14.934 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:38:15.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117046 2024-04-12T16:38:15.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117046 2024-04-12T16:38:15.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863 1-55834574919 2-81604378689 3-115964117046' 2024-04-12T16:38:15.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:15.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:38:15.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757294 2024-04-12T16:38:15.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757294 2024-04-12T16:38:15.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863 1-55834574919 2-81604378689 3-115964117046 4-163208757294' 2024-04-12T16:38:15.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:15.276 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:38:15.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462951 2024-04-12T16:38:15.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462951 2024-04-12T16:38:15.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863 1-55834574919 2-81604378689 3-115964117046 4-163208757294 5-201863462951' 2024-04-12T16:38:15.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:15.561 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:38:15.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168609 2024-04-12T16:38:15.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168609 2024-04-12T16:38:15.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863 1-55834574919 2-81604378689 3-115964117046 4-163208757294 5-201863462951 6-240518168609' 2024-04-12T16:38:15.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:15.719 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:38:15.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841554 2024-04-12T16:38:15.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841554 2024-04-12T16:38:15.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863 1-55834574919 2-81604378689 3-115964117046 4-163208757294 5-201863462951 6-240518168609 7-283467841554' 2024-04-12T16:38:15.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:15.893 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:38:16.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514510 2024-04-12T16:38:16.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514510 2024-04-12T16:38:16.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803863 1-55834574919 2-81604378689 3-115964117046 4-163208757294 5-201863462951 6-240518168609 7-283467841554 8-326417514510' 2024-04-12T16:38:16.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:16.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803863 2024-04-12T16:38:16.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:16.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:38:16.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803863 2024-04-12T16:38:16.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:16.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803863 2024-04-12T16:38:16.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803863' 2024-04-12T16:38:16.052 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803863 2024-04-12T16:38:16.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:38:16.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803863 -lt 25769803863 2024-04-12T16:38:16.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:16.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574919 2024-04-12T16:38:16.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:16.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:38:16.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574919 2024-04-12T16:38:16.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:16.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574919 2024-04-12T16:38:16.474 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574919 2024-04-12T16:38:16.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574919' 2024-04-12T16:38:16.474 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:38:16.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574920 -lt 55834574919 2024-04-12T16:38:16.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:16.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378689 2024-04-12T16:38:16.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:16.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:38:16.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378689 2024-04-12T16:38:16.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:16.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378689 2024-04-12T16:38:16.946 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378689 2024-04-12T16:38:16.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378689' 2024-04-12T16:38:16.946 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:38:17.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378689 -lt 81604378689 2024-04-12T16:38:17.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:17.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117046 2024-04-12T16:38:17.450 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:17.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:38:17.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117046 2024-04-12T16:38:17.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:17.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117046 2024-04-12T16:38:17.454 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 115964117046 2024-04-12T16:38:17.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117046' 2024-04-12T16:38:17.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:38:17.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117046 -lt 115964117046 2024-04-12T16:38:17.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:17.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757294 2024-04-12T16:38:17.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:17.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:38:17.887 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757294 2024-04-12T16:38:17.887 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:17.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757294 2024-04-12T16:38:17.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757294' 2024-04-12T16:38:17.889 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 163208757294 2024-04-12T16:38:17.889 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:38:18.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757295 -lt 163208757294 2024-04-12T16:38:18.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:18.346 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-201863462951 2024-04-12T16:38:18.346 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:18.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:38:18.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-201863462951 2024-04-12T16:38:18.349 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:18.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462951 2024-04-12T16:38:18.350 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 201863462951 2024-04-12T16:38:18.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 201863462951' 2024-04-12T16:38:18.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:38:18.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462951 -lt 201863462951 2024-04-12T16:38:18.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:18.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:18.800 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-240518168609 2024-04-12T16:38:18.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:38:18.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-240518168609 2024-04-12T16:38:18.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:18.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168609 2024-04-12T16:38:18.804 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 240518168609 2024-04-12T16:38:18.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 240518168609' 2024-04-12T16:38:18.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:38:19.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168609 -lt 240518168609 2024-04-12T16:38:19.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:19.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-283467841554 2024-04-12T16:38:19.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:19.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-12T16:38:19.208 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-283467841554 2024-04-12T16:38:19.208 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:19.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841554 2024-04-12T16:38:19.209 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.7 seq 283467841554 2024-04-12T16:38:19.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 283467841554' 2024-04-12T16:38:19.210 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-12T16:38:19.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841555 -lt 283467841554 2024-04-12T16:38:19.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:19.659 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-326417514510 2024-04-12T16:38:19.659 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:19.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:38:19.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-326417514510 2024-04-12T16:38:19.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:19.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514510 2024-04-12T16:38:19.663 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.8 seq 326417514510 2024-04-12T16:38:19.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 326417514510' 2024-04-12T16:38:19.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:38:20.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514511 -lt 326417514510 2024-04-12T16:38:20.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:38:20.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:38:20.063 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:38:20.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:38:20.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:38:20.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:38:20.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:38:20.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:38:20.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:38:20.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:38:20.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:38:21.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:38:21.041 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:38:21.041 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:38:21.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:38:21.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:38:21.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:38:21.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:38:21.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-04-12T16:38:21.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:38:21.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:38:21.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:38:21.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-04-12T16:38:21.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:427: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-04-12T16:38:21.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:38:21.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:38:21.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T16:38:21.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:38:21.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:38:21.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:38:21.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T16:38:21.888 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:38:21.889 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:7' 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-04-12T16:38:22.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:38:22.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-04-12T16:38:22.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:38:22.463 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T16:38:22.464 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T16:38:22.465 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:22.465 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:22.465 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:22.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T16:38:22.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:38:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:38:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:38:22.591 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-04-12T16:38:22.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:38:22.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T16:38:22.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T16:38:22.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:22.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:22.592 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:22.592 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T16:38:22.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:38:22.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-04-12T16:38:22.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-04-12T16:38:22.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:38:22.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:38:22.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T16:38:22.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:38:22.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:38:22.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:38:22.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-04-12T16:38:22.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:38:22.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:38:22.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:38:22.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=3 2024-04-12T16:38:22.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:38:22.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-04-12T16:38:22.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:38:22.714 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T16:38:22.714 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:38:22.714 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:38:23.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:7' 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-04-12T16:38:23.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:38:23.136 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/0/type 2024-04-12T16:38:23.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:38:23.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-12T16:38:23.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:38:23.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-12T16:38:23.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:38:23.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:38:23.139 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:38:23.139 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-12T16:38:23.139 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:38:23.139 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:38:23.139 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:23.139 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:23.139 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:23.139 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T16:38:23.140 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:38:23.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:38:23.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:38:23.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:38:23.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-04-12T16:38:23.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:38:23.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:38:23.263 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:23.263 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:23.264 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:23.264 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T16:38:23.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:38:23.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-04-12T16:38:23.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-04-12T16:38:23.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=4 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:38:23.384 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T16:38:23.385 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:38:23.385 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:38:23.854 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:38:23.854 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:38:23.854 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:38:23.854 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:38:23.854 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:38:23.854 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:38:23.854 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:38:23.855 INFO:tasks.workunit.client.0.smithi137.stderr:7' 2024-04-12T16:38:23.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-04-12T16:38:23.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-04-12T16:38:23.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:38:23.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=6 2024-04-12T16:38:23.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:38:23.855 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/6/type 2024-04-12T16:38:23.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:38:23.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-04-12T16:38:23.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:38:23.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=6 2024-04-12T16:38:23.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:38:23.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:38:23.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:38:23.858 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.6 2024-04-12T16:38:23.858 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-04-12T16:38:23.858 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-04-12T16:38:23.858 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:23.858 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:23.858 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:23.859 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.6.asok 2024-04-12T16:38:23.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:38:23.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:38:23.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:38:23.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:38:23.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.6 2024-04-12T16:38:23.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-04-12T16:38:23.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-04-12T16:38:23.978 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:23.979 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:23.979 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:23.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.6.asok 2024-04-12T16:38:23.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:38:23.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-04-12T16:38:24.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-04-12T16:38:24.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:38:24.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T16:38:24.096 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:38:24.096 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:38:24.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:38:24.521 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:38:24.521 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:7' 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:434: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-04-12T16:38:24.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-04-12T16:38:24.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:38:24.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:38:24.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:38:24.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:38:24.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:38:25.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:38:25.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-04-12T16:38:26.701 INFO:tasks.workunit.client.0.smithi137.stderr:osd.7 is already down. 2024-04-12T16:38:26.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-04-12T16:38:27.614 INFO:tasks.workunit.client.0.smithi137.stderr:osd.7 is already out. 2024-04-12T16:38:27.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-04-12T16:38:27.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:38:27.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:38:27.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:38:27.634 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:38:27.634 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:38:27.635 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:38:27.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:38:27.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:38:27.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:38:27.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:38:27.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:38:27.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:38:27.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:38:27.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:38:27.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:38:28.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:6 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:7 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:8' 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:38:28.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:28.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:38:28.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803867 2024-04-12T16:38:28.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803867 2024-04-12T16:38:28.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803867' 2024-04-12T16:38:28.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:28.765 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:38:29.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574924 2024-04-12T16:38:29.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574924 2024-04-12T16:38:29.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803867 1-55834574924' 2024-04-12T16:38:29.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:29.122 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:38:29.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378694 2024-04-12T16:38:29.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378694 2024-04-12T16:38:29.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803867 1-55834574924 2-81604378694' 2024-04-12T16:38:29.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:29.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:38:29.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117051 2024-04-12T16:38:29.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117051 2024-04-12T16:38:29.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803867 1-55834574924 2-81604378694 3-115964117051' 2024-04-12T16:38:29.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:29.899 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:38:30.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757299 2024-04-12T16:38:30.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757299 2024-04-12T16:38:30.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803867 1-55834574924 2-81604378694 3-115964117051 4-163208757299' 2024-04-12T16:38:30.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:30.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:38:30.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462956 2024-04-12T16:38:30.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462956 2024-04-12T16:38:30.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803867 1-55834574924 2-81604378694 3-115964117051 4-163208757299 5-201863462956' 2024-04-12T16:38:30.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:30.339 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:38:30.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168614 2024-04-12T16:38:30.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168614 2024-04-12T16:38:30.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803867 1-55834574924 2-81604378694 3-115964117051 4-163208757299 5-201863462956 6-240518168614' 2024-04-12T16:38:30.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:30.574 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-12T16:38:30.753 INFO:tasks.workunit.client.0.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-12T16:38:30.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-04-12T16:38:30.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-04-12T16:38:30.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-04-12T16:38:30.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:38:30.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-12T16:38:31.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514515 2024-04-12T16:38:31.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514515 2024-04-12T16:38:31.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803867 1-55834574924 2-81604378694 3-115964117051 4-163208757299 5-201863462956 6-240518168614 8-326417514515' 2024-04-12T16:38:31.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:31.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:31.129 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803867 2024-04-12T16:38:31.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:38:31.131 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803867 2024-04-12T16:38:31.132 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:31.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803867 2024-04-12T16:38:31.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803867' 2024-04-12T16:38:31.133 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803867 2024-04-12T16:38:31.133 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:38:31.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803868 -lt 25769803867 2024-04-12T16:38:31.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:31.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574924 2024-04-12T16:38:31.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:31.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:38:31.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574924 2024-04-12T16:38:31.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:31.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574924 2024-04-12T16:38:31.564 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574924 2024-04-12T16:38:31.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574924' 2024-04-12T16:38:31.565 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:38:32.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574925 -lt 55834574924 2024-04-12T16:38:32.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:32.011 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:32.011 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378694 2024-04-12T16:38:32.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:38:32.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378694 2024-04-12T16:38:32.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:32.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378694 2024-04-12T16:38:32.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378694' 2024-04-12T16:38:32.015 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378694 2024-04-12T16:38:32.015 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:38:32.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378695 -lt 81604378694 2024-04-12T16:38:32.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:32.400 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117051 2024-04-12T16:38:32.401 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:32.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:38:32.403 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117051 2024-04-12T16:38:32.403 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:32.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117051 2024-04-12T16:38:32.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117051' 2024-04-12T16:38:32.405 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 115964117051 2024-04-12T16:38:32.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:38:32.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117051 -lt 115964117051 2024-04-12T16:38:32.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:32.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757299 2024-04-12T16:38:32.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:32.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:38:32.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:32.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757299 2024-04-12T16:38:32.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757299 2024-04-12T16:38:32.869 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 163208757299 2024-04-12T16:38:32.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757299' 2024-04-12T16:38:32.869 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:38:33.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757300 -lt 163208757299 2024-04-12T16:38:33.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:33.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-201863462956 2024-04-12T16:38:33.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:33.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:38:33.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-201863462956 2024-04-12T16:38:33.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:33.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462956 2024-04-12T16:38:33.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 201863462956' 2024-04-12T16:38:33.378 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 201863462956 2024-04-12T16:38:33.379 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:38:33.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462956 -lt 201863462956 2024-04-12T16:38:33.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:33.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-240518168614 2024-04-12T16:38:33.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:33.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:38:33.831 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-240518168614 2024-04-12T16:38:33.831 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:33.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168614 2024-04-12T16:38:33.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 240518168614' 2024-04-12T16:38:33.833 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 240518168614 2024-04-12T16:38:33.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:38:34.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168614 -lt 240518168614 2024-04-12T16:38:34.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:38:34.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-326417514515 2024-04-12T16:38:34.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:38:34.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-12T16:38:34.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-326417514515 2024-04-12T16:38:34.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:38:34.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514515 2024-04-12T16:38:34.264 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.8 seq 326417514515 2024-04-12T16:38:34.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 326417514515' 2024-04-12T16:38:34.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-12T16:38:34.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514516 -lt 326417514515 2024-04-12T16:38:34.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:38:34.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:38:34.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:38:35.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:38:35.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:38:35.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:38:35.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:38:35.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:38:35.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:38:35.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:38:35.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:38:35.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:38:35.645 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:38:35.646 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:38:35.646 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:38:36.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T16:38:36.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-04-12T16:38:36.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T16:38:36.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T16:38:36.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:445: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-04-12T16:38:36.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:38:36.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T16:38:36.877 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T16:38:36.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-04-12T16:38:37.589 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile myprofile does not exist 2024-04-12T16:38:37.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-04-12T16:38:37.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:38:37.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:38:37.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:38:37.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:38:37.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:38:37.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:38:37.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:38:37.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:38:37.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:38:37.917 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:38:37.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:38:37.918 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:38:37.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:38:37.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:38:37.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:38:37.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:38:37.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:38:37.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:38:37.921 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:38:37.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:38:37.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:38:37.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:38:37.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:38:37.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:37.983 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:37.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:38:37.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:38:37.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:38:37.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:38:37.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:38:37.989 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:38:37.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:38:37.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:38:37.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:38:37.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:38:37.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:38:37.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:38:37.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:38:37.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:38:37.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:38:37.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:38:37.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:38:37.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:38:37.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:38:37.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:37.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:37.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:38:37.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:38:37.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:38:37.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T16:38:38.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:38:38.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:38.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:38.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T16:38:38.002 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:38:38.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:38:38.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:38:38.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T16:38:38.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T16:38:38.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T16:38:38.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:38:38.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:38:38.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:38:38.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T16:38:38.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T16:38:38.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:38:38.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:38:38.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:38:38.526 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:38:38.526 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:38.526 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:38.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:38:38.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:38:38.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:38:38.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:38:38.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:38:38.762 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:38:38.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:38:38.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:38:38.764 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:38:38.764 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:38:38.764 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:38:38.764 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:38.764 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:38.764 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:38.764 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:38:38.764 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:38:38.765 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T16:38:38.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:38:38.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:38:38.879 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:38:38.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:38:38.880 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:38:38.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:38:38.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:38:38.880 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:38:38.881 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:38.881 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:38.881 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:38.881 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:38:38.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:38:38.881 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T16:38:38.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T16:38:38.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T16:38:38.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:38:38.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:38:38.983 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:38:38.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T16:38:38.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:38:39.926 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:38:39.926 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:38:39.926 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:38:39.926 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:38:39.926 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:39.926 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:39.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:38:39.927 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:38:39.929 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:38:39.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T16:38:39.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:38:40.323 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:38:40.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:38:41.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T16:38:41.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:38:41.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:38:41.345 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:38:41.345 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:41.345 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:41.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:38:41.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T16:38:41.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T16:38:41.445 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:38:41.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T16:38:41.461 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:38:38.570+0000 7fd6f2e6fc80 0 load: jerasure load: lrc 2024-04-12T16:38:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-04-12T16:38:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-04-12T16:38:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:452: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-04-12T16:38:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:38:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:38:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-04-12T16:38:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-04-12T16:38:41.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T16:38:41.463 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-04-12T16:38:41.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-04-12T16:38:41.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T16:38:41.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:38:41.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a442401e-d81a-43c9-bcd5-6efb0abb6b85 2024-04-12T16:38:41.472 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 a442401e-d81a-43c9-bcd5-6efb0abb6b85 2024-04-12T16:38:41.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a442401e-d81a-43c9-bcd5-6efb0abb6b85' 2024-04-12T16:38:41.473 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:38:41.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRYxlmCsiIHRAA501MmvVrVsY/yDqO0dNtbw== 2024-04-12T16:38:41.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRYxlmCsiIHRAA501MmvVrVsY/yDqO0dNtbw=="}' 2024-04-12T16:38:41.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a442401e-d81a-43c9-bcd5-6efb0abb6b85 -i td/test-erasure-eio/0/new.json 2024-04-12T16:38:41.857 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:38:41.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T16:38:41.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRYxlmCsiIHRAA501MmvVrVsY/yDqO0dNtbw== --osd-uuid a442401e-d81a-43c9-bcd5-6efb0abb6b85 2024-04-12T16:38:41.913 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:41.914+0000 7f9ab19c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:38:41.942 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:41.946+0000 7f9ab19c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:38:41.944 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:41.946+0000 7f9ab19c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:38:41.944 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:41.946+0000 7f9ab19c57c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:38:41.944 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:41.946+0000 7f9ab19c57c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:38:41.944 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:41.946+0000 7f9ab19c57c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:38:41.991 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:41.994+0000 7f9ab19c57c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T16:38:49.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T16:38:49.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:38:49.820 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:38:49.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:38:49.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:38:50.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:38:50.391 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:38:50.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:38:50.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:38:50.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:38:50.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:38:50.430 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:50.430+0000 7f22490737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:38:50.430 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:50.434+0000 7f22490737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:38:50.430 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:50.434+0000 7f22490737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:38:50.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:38:50.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:38:50.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:38:50.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:38:50.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:38:50.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:50.795 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:38:50.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:38:50.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:50.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:38:50.999 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:51.002+0000 7f22490737c0 -1 Falling back to public interface 2024-04-12T16:38:51.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:52.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:52.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:52.246 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:38:52.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:38:52.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:52.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:38:52.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:53.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:53.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:53.660 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:38:53.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:38:53.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:53.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:38:53.779 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:38:53.782+0000 7f22490737c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:38:54.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:55.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:55.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:55.166 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:38:55.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:38:55.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:55.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:38:55.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:56.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:56.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:56.812 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:38:56.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:38:56.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:56.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:38:57.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:58.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:58.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:58.324 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:38:58.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:38:58.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:58.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:38:58.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:38:59.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:38:59.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:38:59.830 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:38:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:38:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:38:59.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:00.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:01.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:01.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:01.299 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:39:01.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:39:01.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:01.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:01.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:02.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:02.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:02.839 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:39:02.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:39:02.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:02.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:03.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:04.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:04.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:04.291 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:39:04.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:39:04.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:04.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:04.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:05.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:05.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:05.900 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:39:05.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:39:05.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:05.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:06.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:07.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:07.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:07.477 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:39:07.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:39:07.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:07.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:08.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:09.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:09.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:09.149 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:39:09.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:39:09.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:09.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:09.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:10.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:10.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:10.823 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:39:10.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:39:10.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:10.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:11.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:12.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:12.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:12.344 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:39:12.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:39:12.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:12.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:13.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:14.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:14.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:14.217 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:39:14.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:39:14.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:14.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:14.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:15.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:15.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:15.995 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:39:15.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:39:15.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:15.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:16.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:17.865 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:39:17.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:17.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:17.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:39:17.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:17.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:18.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:19.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:19.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:19.743 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:39:19.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:39:19.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:19.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:20.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:21.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:21.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:21.631 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:39:21.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:39:21.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:21.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:22.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:23.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:23.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:23.430 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:39:23.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:39:23.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:23.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:24.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:25.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:25.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:25.074 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:39:25.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:39:25.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:25.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:39:25.613 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1211074278,v1:127.0.0.1:6803/1211074278] [v2:127.0.0.1:6804/1211074278,v1:127.0.0.1:6805/1211074278] exists,up a442401e-d81a-43c9-bcd5-6efb0abb6b85 2024-04-12T16:39:25.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:39:25.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T16:39:25.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:39:25.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T16:39:25.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:39:25.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d3a5f3f-b470-4137-b9aa-3df69e2cf03b 2024-04-12T16:39:25.620 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 8d3a5f3f-b470-4137-b9aa-3df69e2cf03b 2024-04-12T16:39:25.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8d3a5f3f-b470-4137-b9aa-3df69e2cf03b' 2024-04-12T16:39:25.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:39:25.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9Yxlm0k9xJhAAVuYTgZOJFRi+JFCP+OKHkA== 2024-04-12T16:39:25.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9Yxlm0k9xJhAAVuYTgZOJFRi+JFCP+OKHkA=="}' 2024-04-12T16:39:25.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d3a5f3f-b470-4137-b9aa-3df69e2cf03b -i td/test-erasure-eio/1/new.json 2024-04-12T16:39:26.471 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:39:26.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T16:39:26.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9Yxlm0k9xJhAAVuYTgZOJFRi+JFCP+OKHkA== --osd-uuid 8d3a5f3f-b470-4137-b9aa-3df69e2cf03b 2024-04-12T16:39:26.529 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:26.534+0000 7f8533bb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:39:26.559 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:26.562+0000 7f8533bb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:39:26.561 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:26.566+0000 7f8533bb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:39:26.561 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:26.566+0000 7f8533bb37c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:39:26.561 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:26.566+0000 7f8533bb37c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:39:26.562 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:26.566+0000 7f8533bb37c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:39:26.670 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:26.670+0000 7f8533bb37c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T16:39:34.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T16:39:34.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:39:34.475 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:39:34.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:39:34.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:39:35.032 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:39:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:39:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:39:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:39:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:39:35.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:39:35.072 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:35.074+0000 7fd85e8be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:39:35.074 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:35.078+0000 7fd85e8be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:39:35.075 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:35.078+0000 7fd85e8be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:39:35.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:39:35.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:39:35.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:39:35.457 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:39:35.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:39:35.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:39:35.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:35.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:39:35.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:35.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:35.630 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:35.634+0000 7fd85e8be7c0 -1 Falling back to public interface 2024-04-12T16:39:35.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:36.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:36.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:36.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:39:36.854 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:39:36.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:36.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:37.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:38.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:38.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:38.278 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:39:38.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:39:38.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:38.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:38.373 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:39:38.374+0000 7fd85e8be7c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:39:38.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:39.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:39.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:39.828 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:39:39.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:39:39.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:39.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:40.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:41.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:41.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:41.264 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:39:41.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:39:41.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:41.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:41.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:42.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:42.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:42.961 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:39:42.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:39:42.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:42.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:43.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:44.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:44.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:44.502 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:39:44.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:39:44.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:44.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:44.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:45.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:45.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:45.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:39:45.989 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:39:45.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:45.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:46.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:47.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:47.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:47.571 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:39:47.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:39:47.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:47.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:48.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:49.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:49.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:49.054 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:39:49.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:39:49.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:49.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:49.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:50.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:50.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:50.485 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:39:50.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:39:50.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:50.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:51.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:52.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:52.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:52.046 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:39:52.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:39:52.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:52.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:52.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:53.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:53.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:53.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:39:53.654 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:39:53.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:53.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:54.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:55.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:55.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:55.203 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:39:55.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:39:55.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:55.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:55.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:56.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:56.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:56.959 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:39:56.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:39:56.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:56.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:58.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:39:59.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:39:59.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:39:59.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:39:59.188 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:39:59.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:39:59.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:39:59.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:00.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:00.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:00.883 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:40:00.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:40:00.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:00.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:40:01.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:02.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:02.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:02.823 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:40:02.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:40:02.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:02.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:40:04.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:05.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:05.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:05.210 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:40:05.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:40:05.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:05.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:40:06.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:07.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:07.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:07.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:40:07.207 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:40:07.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:07.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:40:08.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:09.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:09.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:09.457 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:40:09.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:40:09.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:09.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:40:10.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:11.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:11.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:11.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:40:11.018 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:40:11.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:11.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:40:11.460 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2487385662,v1:127.0.0.1:6811/2487385662] [v2:127.0.0.1:6812/2487385662,v1:127.0.0.1:6813/2487385662] exists,up 8d3a5f3f-b470-4137-b9aa-3df69e2cf03b 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:40:11.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:40:11.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:40:11.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:40:11.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:40:11.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:40:11.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:40:11.467 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:40:11.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d123c89b-dcf5-4574-9e30-873a7f822ac7 2024-04-12T16:40:11.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d123c89b-dcf5-4574-9e30-873a7f822ac7' 2024-04-12T16:40:11.468 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 d123c89b-dcf5-4574-9e30-873a7f822ac7 2024-04-12T16:40:11.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:40:11.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrYxlmQzyxHRAAQ+bfhTnAWX8hlQ5YTTJkJQ== 2024-04-12T16:40:11.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrYxlmQzyxHRAAQ+bfhTnAWX8hlQ5YTTJkJQ=="}' 2024-04-12T16:40:11.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d123c89b-dcf5-4574-9e30-873a7f822ac7 -i td/test-erasure-eio/2/new.json 2024-04-12T16:40:12.230 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:40:12.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T16:40:12.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrYxlmQzyxHRAAQ+bfhTnAWX8hlQ5YTTJkJQ== --osd-uuid d123c89b-dcf5-4574-9e30-873a7f822ac7 2024-04-12T16:40:12.287 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:12.290+0000 7f7527e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:12.418 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:12.422+0000 7f7527e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:12.420 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:12.422+0000 7f7527e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:12.421 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:12.426+0000 7f7527e2b7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:40:12.421 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:12.426+0000 7f7527e2b7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:40:12.421 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:12.426+0000 7f7527e2b7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:40:12.464 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:12.466+0000 7f7527e2b7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T16:40:20.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T16:40:20.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:40:20.436 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:40:20.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:40:20.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:40:20.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:40:20.999 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:40:20.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:40:20.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:40:21.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:40:21.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:40:21.035 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:21.038+0000 7f225a35c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:21.071 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:21.074+0000 7f225a35c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:21.073 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:21.078+0000 7f225a35c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:21.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:40:21.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:40:21.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:40:21.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:40:21.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:40:21.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:21.457 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:40:21.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:40:21.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:21.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:21.626 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:21.630+0000 7f225a35c7c0 -1 Falling back to public interface 2024-04-12T16:40:21.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:22.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:22.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:22.914 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:40:22.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:40:22.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:22.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:23.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:24.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:24.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:24.389 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:40:24.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:40:24.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:24.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:24.467 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:24.470+0000 7f225a35c7c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:40:24.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:25.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:25.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:25.925 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:40:25.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:40:25.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:25.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:26.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:27.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:27.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:27.348 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:40:27.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:40:27.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:27.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:28.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:29.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:29.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:29.057 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:40:29.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:40:29.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:29.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:29.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:30.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:30.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:30.790 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:40:30.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:40:30.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:30.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:31.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:32.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:32.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:32.389 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:40:32.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:40:32.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:32.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:32.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:33.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:33.995 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:40:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:40:33.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:33.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:34.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:35.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:35.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:35.641 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:40:35.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:40:35.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:35.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:36.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:37.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:37.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:37.276 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:40:37.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:40:37.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:37.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:37.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:38.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:38.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:38.798 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:40:38.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:40:38.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:38.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:39.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:40.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:40.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:40.268 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:40:40.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:40:40.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:40.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:40.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:41.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:41.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:41.953 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:40:41.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:40:41.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:41.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:42.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:43.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:43.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:43.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:40:43.558 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:40:43.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:43.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:44.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:45.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:45.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:45.605 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:40:45.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:40:45.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:45.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:46.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:47.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:47.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:47.645 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:40:47.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:40:47.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:47.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:48.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:49.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:49.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:49.687 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:40:49.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:40:49.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:49.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:50.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:51.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:51.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:51.697 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:40:51.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:40:51.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:51.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:52.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:53.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:53.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:53.388 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:40:53.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:40:53.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:53.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:54.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:55.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:55.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:55.128 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:40:55.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:40:55.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:55.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:56.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:40:57.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:40:57.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:40:57.147 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:40:57.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:40:57.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:40:57.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:40:57.664 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2690220765,v1:127.0.0.1:6819/2690220765] [v2:127.0.0.1:6820/2690220765,v1:127.0.0.1:6821/2690220765] exists,up d123c89b-dcf5-4574-9e30-873a7f822ac7 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:40:57.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:40:57.668 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:40:57.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:40:57.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:40:57.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:40:57.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:40:57.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:40:57.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:40:57.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:40:57.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:40:57.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:40:57.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:40:57.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:40:57.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:40:57.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:40:57.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T16:40:57.671 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:40:57.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5bcd9bc1-5fb5-4eb6-81d4-ff00dbd9866a 2024-04-12T16:40:57.672 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 5bcd9bc1-5fb5-4eb6-81d4-ff00dbd9866a 2024-04-12T16:40:57.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5bcd9bc1-5fb5-4eb6-81d4-ff00dbd9866a' 2024-04-12T16:40:57.672 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:40:57.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZZBlmwAmdKRAApksnZRdA/WagjlTSPeOpdA== 2024-04-12T16:40:57.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZZBlmwAmdKRAApksnZRdA/WagjlTSPeOpdA=="}' 2024-04-12T16:40:57.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5bcd9bc1-5fb5-4eb6-81d4-ff00dbd9866a -i td/test-erasure-eio/3/new.json 2024-04-12T16:40:59.012 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:40:59.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T16:40:59.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZZBlmwAmdKRAApksnZRdA/WagjlTSPeOpdA== --osd-uuid 5bcd9bc1-5fb5-4eb6-81d4-ff00dbd9866a 2024-04-12T16:40:59.077 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:59.082+0000 7f8bf06d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:59.080 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:59.082+0000 7f8bf06d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:59.081 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:59.086+0000 7f8bf06d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:40:59.082 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:59.086+0000 7f8bf06d27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:40:59.082 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:59.086+0000 7f8bf06d27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:40:59.082 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:59.086+0000 7f8bf06d27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:40:59.241 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:40:59.246+0000 7f8bf06d27c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T16:41:04.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T16:41:04.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:41:04.814 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T16:41:04.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T16:41:04.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:41:05.379 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:41:05.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T16:41:05.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:41:05.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:41:05.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:41:05.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:41:05.409 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:05.414+0000 7f0b74e867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:05.411 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:05.414+0000 7f0b74e867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:05.412 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:05.414+0000 7f0b74e867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:05.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T16:41:05.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:41:05.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:41:05.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:41:05.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:41:05.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:05.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:41:05.812 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:41:05.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:05.812 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:05.954 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:05.958+0000 7f0b74e867c0 -1 Falling back to public interface 2024-04-12T16:41:06.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:07.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:07.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:07.315 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:41:07.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:41:07.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:07.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:07.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:08.379 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:08.382+0000 7f0b74e867c0 -1 osd.3 0 log_to_monitors true 2024-04-12T16:41:08.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:08.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:08.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:41:08.725 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:41:08.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:08.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:09.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:10.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:10.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:10.331 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:41:10.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:41:10.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:10.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:10.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:11.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:11.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:11.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:41:11.954 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:41:11.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:11.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:12.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:13.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:13.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:13.442 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:41:13.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:41:13.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:13.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:13.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:14.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:14.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:14.910 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:41:14.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:41:14.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:14.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:15.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:16.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:16.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:16.466 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:41:16.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:41:16.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:16.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:16.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:17.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:17.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:17.889 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:41:17.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:41:17.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:17.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:18.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:19.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:19.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:19.450 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:41:19.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:41:19.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:19.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:20.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:21.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:21.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:21.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:41:21.004 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:41:21.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:21.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:21.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:22.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:22.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:22.494 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:41:22.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:41:22.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:22.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:23.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:24.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:24.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:24.040 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:41:24.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:41:24.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:24.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:25.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:26.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:26.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:26.290 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:41:26.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:41:26.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:26.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:27.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:28.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:28.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:28.169 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:41:28.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:41:28.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:28.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:29.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:30.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:30.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:30.163 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:41:30.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:41:30.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:30.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:30.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:31.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:31.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:31.952 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:41:31.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:41:31.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:31.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:32.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:33.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:33.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:33.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:41:33.556 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:41:33.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:33.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:34.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:35.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:35.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:35.087 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:41:35.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:41:35.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:35.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:41:35.690 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3543481652,v1:127.0.0.1:6827/3543481652] [v2:127.0.0.1:6828/3543481652,v1:127.0.0.1:6829/3543481652] exists,up 5bcd9bc1-5fb5-4eb6-81d4-ff00dbd9866a 2024-04-12T16:41:35.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:41:35.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:41:35.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:41:35.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-04-12T16:41:35.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:41:35.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:41:35.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-04-12T16:41:35.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:41:35.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2fea3b86-4e54-4643-95ec-765c4717045c 2024-04-12T16:41:35.697 INFO:tasks.workunit.client.0.smithi137.stdout:add osd4 2fea3b86-4e54-4643-95ec-765c4717045c 2024-04-12T16:41:35.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 2fea3b86-4e54-4643-95ec-765c4717045c' 2024-04-12T16:41:35.697 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:41:35.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/ZBlmCRhLKxAAo4MHpBBaygB3M3Gr3cATFw== 2024-04-12T16:41:35.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/ZBlmCRhLKxAAo4MHpBBaygB3M3Gr3cATFw=="}' 2024-04-12T16:41:35.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2fea3b86-4e54-4643-95ec-765c4717045c -i td/test-erasure-eio/4/new.json 2024-04-12T16:41:36.803 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:41:36.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-04-12T16:41:36.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/ZBlmCRhLKxAAo4MHpBBaygB3M3Gr3cATFw== --osd-uuid 2fea3b86-4e54-4643-95ec-765c4717045c 2024-04-12T16:41:36.866 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:36.870+0000 7fe04c8537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:36.872 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:36.874+0000 7fe04c8537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:36.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:36.878+0000 7fe04c8537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:36.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:36.878+0000 7fe04c8537c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:41:36.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:36.878+0000 7fe04c8537c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:41:36.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:36.878+0000 7fe04c8537c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:41:37.176 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:37.182+0000 7fe04c8537c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-04-12T16:41:44.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-04-12T16:41:44.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:41:44.885 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd4 key to auth repository 2024-04-12T16:41:44.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-12T16:41:44.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:41:45.457 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.4 2024-04-12T16:41:45.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-12T16:41:45.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:41:45.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:41:45.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:41:45.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:41:45.487 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:45.490+0000 7efcda8627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:45.490 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:45.494+0000 7efcda8627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:45.491 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:45.494+0000 7efcda8627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:41:45.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-12T16:41:45.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:41:45.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-12T16:41:45.877 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:41:45.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:41:45.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:41:45.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:45.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:41:45.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:45.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:46.054 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:46.058+0000 7efcda8627c0 -1 Falling back to public interface 2024-04-12T16:41:46.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:47.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:47.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:47.283 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:41:47.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:41:47.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:47.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:47.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:48.570 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:41:48.574+0000 7efcda8627c0 -1 osd.4 0 log_to_monitors true 2024-04-12T16:41:48.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:48.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:48.702 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:41:48.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:41:48.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:48.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:49.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:50.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:50.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:50.223 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:41:50.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:41:50.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:50.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:51.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:52.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:52.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:52.450 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:41:52.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:41:52.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:52.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:53.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:54.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:54.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:54.507 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:41:54.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:41:54.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:54.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:55.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:56.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:41:56.004 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:41:56.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:56.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:56.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:57.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:57.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:57.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:41:57.593 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:41:57.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:57.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:58.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:41:59.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:41:59.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:41:59.036 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:41:59.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:41:59.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:41:59.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:41:59.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:00.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:00.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:00.621 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:42:00.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:42:00.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:00.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:01.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:02.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:02.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:02.111 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:42:02.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:42:02.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:02.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:02.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:03.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:03.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:03.653 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:42:03.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:42:03.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:03.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:04.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:05.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:05.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:05.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:42:05.259 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:42:05.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:05.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:06.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:07.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:07.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:07.367 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:42:07.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:42:07.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:07.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:08.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:09.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:09.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:09.238 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:42:09.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:42:09.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:09.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:09.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:10.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:10.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:10.718 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:42:10.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:42:10.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:10.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:11.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:12.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:12.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:12.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:42:12.179 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:42:12.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:12.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:12.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:13.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:13.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:13.659 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:42:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:42:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:13.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:14.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:15.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:15.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:15.171 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:42:15.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:42:15.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:15.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:42:15.694 INFO:tasks.workunit.client.0.smithi137.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1491269525,v1:127.0.0.1:6835/1491269525] [v2:127.0.0.1:6836/1491269525,v1:127.0.0.1:6837/1491269525] exists,up 2fea3b86-4e54-4643-95ec-765c4717045c 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-04-12T16:42:15.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:42:15.699 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:42:15.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:42:15.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:42:15.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:42:15.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:42:15.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-04-12T16:42:15.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:42:15.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3abd8c1-3cf4-4fb5-bf18-0dbcfd35387a 2024-04-12T16:42:15.701 INFO:tasks.workunit.client.0.smithi137.stdout:add osd5 f3abd8c1-3cf4-4fb5-bf18-0dbcfd35387a 2024-04-12T16:42:15.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 f3abd8c1-3cf4-4fb5-bf18-0dbcfd35387a' 2024-04-12T16:42:15.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:42:15.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnZBlmJ/tzKxAAspB2KgUYEsBHCZ5Ftuc4Fw== 2024-04-12T16:42:15.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnZBlmJ/tzKxAAspB2KgUYEsBHCZ5Ftuc4Fw=="}' 2024-04-12T16:42:15.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3abd8c1-3cf4-4fb5-bf18-0dbcfd35387a -i td/test-erasure-eio/5/new.json 2024-04-12T16:42:16.795 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:42:16.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-04-12T16:42:16.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnZBlmJ/tzKxAAspB2KgUYEsBHCZ5Ftuc4Fw== --osd-uuid f3abd8c1-3cf4-4fb5-bf18-0dbcfd35387a 2024-04-12T16:42:16.856 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:16.858+0000 7f99b3b207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:42:16.901 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:16.906+0000 7f99b3b207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:42:16.903 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:16.906+0000 7f99b3b207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:42:16.903 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:16.906+0000 7f99b3b207c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:42:16.904 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:16.906+0000 7f99b3b207c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:42:16.904 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:16.906+0000 7f99b3b207c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:42:17.126 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:17.130+0000 7f99b3b207c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-04-12T16:42:25.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-04-12T16:42:25.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:42:25.541 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd5 key to auth repository 2024-04-12T16:42:25.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-12T16:42:25.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:42:26.118 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.5 2024-04-12T16:42:26.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-12T16:42:26.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:42:26.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:42:26.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:42:26.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:42:26.155 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:26.158+0000 7f53927637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:42:26.200 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:26.202+0000 7f53927637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:42:26.201 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:26.206+0000 7f53927637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:42:26.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-12T16:42:26.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:42:26.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-12T16:42:26.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:42:26.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:42:26.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:26.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:42:26.540 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:42:26.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:26.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:26.763 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:26.766+0000 7f53927637c0 -1 Falling back to public interface 2024-04-12T16:42:26.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:27.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:27.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:27.935 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:42:27.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:42:27.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:27.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:28.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:29.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:29.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:29.375 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:42:29.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:42:29.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:29.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:29.658 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:42:29.662+0000 7f53927637c0 -1 osd.5 0 log_to_monitors true 2024-04-12T16:42:29.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:30.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:30.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:30.804 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:42:30.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:42:30.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:30.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:31.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:32.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:32.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:32.602 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:42:32.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:42:32.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:32.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:33.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:34.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:34.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:34.210 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:42:34.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:42:34.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:34.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:35.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:36.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:36.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:36.030 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:42:36.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:42:36.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:36.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:36.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:37.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:37.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:37.547 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:42:37.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:42:37.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:37.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:38.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:39.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:39.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:39.241 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:42:39.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:42:39.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:39.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:39.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:40.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:40.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:40.781 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:42:40.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:42:40.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:40.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:41.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:42.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:42.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:42.529 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:42:42.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:42:42.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:42.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:43.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:44.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:44.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:44.092 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:42:44.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:42:44.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:44.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:44.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:45.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:45.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:45.787 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:42:45.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:42:45.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:45.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:46.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:47.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:47.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:47.321 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:42:47.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:42:47.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:47.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:48.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:49.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:49.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:49.049 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:42:49.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:42:49.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:49.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:49.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:50.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:50.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:50.908 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:42:50.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:42:50.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:50.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:51.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:52.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:52.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:52.843 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:42:52.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:42:52.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:52.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:53.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:54.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:54.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:54.535 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:42:54.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:42:54.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:54.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:55.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:56.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:56.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:56.425 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:42:56.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:42:56.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:56.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:57.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:42:58.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:42:58.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:42:58.351 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:42:58.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:42:58.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:42:58.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:42:59.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:00.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:00.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:00.143 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:43:00.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:43:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:43:00.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:01.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:01.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:01.889 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:43:01.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:43:01.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:01.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:43:02.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:03.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:03.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:03.464 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:43:03.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:43:03.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:03.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:43:04.033 INFO:tasks.workunit.client.0.smithi137.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2879066541,v1:127.0.0.1:6843/2879066541] [v2:127.0.0.1:6844/2879066541,v1:127.0.0.1:6845/2879066541] exists,up f3abd8c1-3cf4-4fb5-bf18-0dbcfd35387a 2024-04-12T16:43:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:43:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:43:04.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:43:04.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:43:04.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:43:04.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-04-12T16:43:04.041 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:43:04.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=907f8da6-d221-47b6-9604-9f8b81766ed9 2024-04-12T16:43:04.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 907f8da6-d221-47b6-9604-9f8b81766ed9' 2024-04-12T16:43:04.041 INFO:tasks.workunit.client.0.smithi137.stdout:add osd6 907f8da6-d221-47b6-9604-9f8b81766ed9 2024-04-12T16:43:04.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:43:04.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYZBlmNzhCBBAAn9Q1sTGL5ipIugPeEi0Opw== 2024-04-12T16:43:04.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYZBlmNzhCBBAAn9Q1sTGL5ipIugPeEi0Opw=="}' 2024-04-12T16:43:04.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 907f8da6-d221-47b6-9604-9f8b81766ed9 -i td/test-erasure-eio/6/new.json 2024-04-12T16:43:05.100 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:43:05.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-04-12T16:43:05.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYZBlmNzhCBBAAn9Q1sTGL5ipIugPeEi0Opw== --osd-uuid 907f8da6-d221-47b6-9604-9f8b81766ed9 2024-04-12T16:43:05.156 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:05.158+0000 7f7b428217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:43:05.190 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:05.194+0000 7f7b428217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:43:05.191 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:05.194+0000 7f7b428217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:43:05.192 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:05.194+0000 7f7b428217c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:43:05.192 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:05.194+0000 7f7b428217c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:43:05.192 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:05.194+0000 7f7b428217c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:43:05.323 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:05.326+0000 7f7b428217c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-04-12T16:43:13.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-04-12T16:43:13.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:43:13.439 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd6 key to auth repository 2024-04-12T16:43:13.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-12T16:43:13.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:43:14.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-12T16:43:14.037 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.6 2024-04-12T16:43:14.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:43:14.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:43:14.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:43:14.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:43:14.073 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:14.078+0000 7f1c2947f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:43:14.111 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:14.114+0000 7f1c2947f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:43:14.113 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:14.118+0000 7f1c2947f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:43:14.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-12T16:43:14.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:43:14.458 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:43:14.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-12T16:43:14.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:43:14.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:43:14.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:14.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:43:14.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:14.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:14.675 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:14.678+0000 7f1c2947f7c0 -1 Falling back to public interface 2024-04-12T16:43:14.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:15.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:15.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:15.886 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:43:15.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:43:15.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:15.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:16.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:17.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:17.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:17.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:43:17.299 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:43:17.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:17.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:17.628 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:43:17.629+0000 7f1c2947f7c0 -1 osd.6 0 log_to_monitors true 2024-04-12T16:43:17.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:18.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:18.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:18.754 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:43:18.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:43:18.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:18.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:19.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:20.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:20.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:20.500 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:43:20.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:43:20.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:20.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:21.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:22.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:22.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:22.197 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:43:22.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:43:22.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:22.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:23.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:24.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:24.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:24.030 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:43:24.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:43:24.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:24.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:24.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:25.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:25.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:25.642 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:43:25.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:43:25.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:25.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:26.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:27.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:27.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:27.259 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:43:27.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:43:27.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:27.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:27.958 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:28.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:28.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:28.960 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:43:28.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:43:28.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:28.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:29.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:30.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:30.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:30.518 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:43:30.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:43:30.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:30.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:31.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:32.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:32.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:32.152 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:43:32.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:43:32.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:32.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:32.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:33.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:33.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:33.613 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:43:33.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:43:33.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:33.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:34.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:35.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:35.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:35.112 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:43:35.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:43:35.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:35.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:35.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:36.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:36.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:36.817 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:43:36.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:43:36.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:36.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:37.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:38.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:38.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:38.318 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:43:38.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:43:38.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:38.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:39.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:40.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:40.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:40.030 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:43:40.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:43:40.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:40.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:40.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:41.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:41.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:41.893 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:43:41.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:43:41.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:41.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:42.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:43.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:43.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:43.978 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:43:43.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:43:43.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:43.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:44.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:45.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:45.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:45.815 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:43:45.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:43:45.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:45.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:46.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:47.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:47.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:47.621 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:43:47.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:43:47.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:47.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:48.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:49.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:49.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:49.479 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:43:49.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:43:49.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:49.480 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:50.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:43:51.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:43:51.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:43:51.274 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:43:51.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:43:51.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:43:51.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:43:51.739 INFO:tasks.workunit.client.0.smithi137.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1094601460,v1:127.0.0.1:6851/1094601460] [v2:127.0.0.1:6852/1094601460,v1:127.0.0.1:6853/1094601460] exists,up 907f8da6-d221-47b6-9604-9f8b81766ed9 2024-04-12T16:43:51.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:43:51.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:43:51.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:43:51.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T16:43:51.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:43:51.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:43:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:43:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:43:51.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:43:51.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T16:43:51.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T16:43:51.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T16:43:51.846 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:43:51.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T16:43:51.866 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:38:51.006+0000 7f22490737c0 0 load: jerasure load: lrc 2024-04-12T16:43:51.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:43:51.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-04-12T16:43:51.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:460: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-04-12T16:43:51.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:43:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T16:43:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-04-12T16:43:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T16:43:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-04-12T16:43:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T16:43:51.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-04-12T16:43:53.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T16:43:53.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T16:43:54.476 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T16:43:54.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:43:55.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T16:43:55.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:43:55.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:43:55.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:43:55.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:43:55.498 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:43:55.498 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:43:55.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:43:55.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:43:55.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:43:55.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:43:55.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:43:55.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:43:55.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:43:55.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:43:55.622 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:43:56.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:43:56.295 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:43:56.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803835 2024-04-12T16:43:56.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803835 2024-04-12T16:43:56.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835' 2024-04-12T16:43:56.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:43:56.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:43:56.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574905 2024-04-12T16:43:56.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574905 2024-04-12T16:43:56.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574905' 2024-04-12T16:43:56.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:43:56.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:43:56.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378673 2024-04-12T16:43:56.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378673 2024-04-12T16:43:56.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574905 2-81604378673' 2024-04-12T16:43:56.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:43:56.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:43:56.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084320 2024-04-12T16:43:56.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084320 2024-04-12T16:43:56.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574905 2-81604378673 3-120259084320' 2024-04-12T16:43:56.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:43:56.968 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:43:57.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659160 2024-04-12T16:43:57.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659160 2024-04-12T16:43:57.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574905 2-81604378673 3-120259084320 4-176093659160' 2024-04-12T16:43:57.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:43:57.116 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:43:57.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332119 2024-04-12T16:43:57.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332119 2024-04-12T16:43:57.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574905 2-81604378673 3-120259084320 4-176093659160 5-219043332119' 2024-04-12T16:43:57.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:43:57.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:43:57.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037774 2024-04-12T16:43:57.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037774 2024-04-12T16:43:57.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574905 2-81604378673 3-120259084320 4-176093659160 5-219043332119 6-257698037774' 2024-04-12T16:43:57.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:43:57.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803835 2024-04-12T16:43:57.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:43:57.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:43:57.487 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803835 2024-04-12T16:43:57.487 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:43:57.488 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803835 2024-04-12T16:43:57.488 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803835 2024-04-12T16:43:57.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-04-12T16:43:57.489 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:43:57.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803835 -lt 25769803835 2024-04-12T16:43:57.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:43:57.908 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574905 2024-04-12T16:43:57.908 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:43:57.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:43:57.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574905 2024-04-12T16:43:57.910 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:43:57.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574905 2024-04-12T16:43:57.911 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574905 2024-04-12T16:43:57.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574905' 2024-04-12T16:43:57.912 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:43:58.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574905 -lt 55834574905 2024-04-12T16:43:58.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:43:58.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378673 2024-04-12T16:43:58.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:43:58.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:43:58.376 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378673 2024-04-12T16:43:58.376 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:43:58.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378673 2024-04-12T16:43:58.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378673' 2024-04-12T16:43:58.377 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378673 2024-04-12T16:43:58.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:43:58.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378674 -lt 81604378673 2024-04-12T16:43:58.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:43:58.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084320 2024-04-12T16:43:58.804 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:43:58.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:43:58.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084320 2024-04-12T16:43:58.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:43:58.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084320 2024-04-12T16:43:58.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084320' 2024-04-12T16:43:58.808 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 120259084320 2024-04-12T16:43:58.808 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:43:59.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084320 -lt 120259084320 2024-04-12T16:43:59.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:43:59.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:43:59.238 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659160 2024-04-12T16:43:59.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:43:59.240 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659160 2024-04-12T16:43:59.240 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:43:59.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659160 2024-04-12T16:43:59.242 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 176093659160 2024-04-12T16:43:59.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659160' 2024-04-12T16:43:59.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:43:59.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659160 -lt 176093659160 2024-04-12T16:43:59.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:43:59.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332119 2024-04-12T16:43:59.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:43:59.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:43:59.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332119 2024-04-12T16:43:59.630 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:43:59.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332119 2024-04-12T16:43:59.631 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 219043332119 2024-04-12T16:43:59.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332119' 2024-04-12T16:43:59.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:44:00.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332120 -lt 219043332119 2024-04-12T16:44:00.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:44:00.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-257698037774 2024-04-12T16:44:00.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:44:00.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:44:00.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-257698037774 2024-04-12T16:44:00.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:44:00.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037774 2024-04-12T16:44:00.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 257698037774' 2024-04-12T16:44:00.101 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 257698037774 2024-04-12T16:44:00.101 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:44:00.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037774 -lt 257698037774 2024-04-12T16:44:00.541 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:44:00.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:44:00.542 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:44:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:44:01.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:44:01.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:44:01.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:44:01.105 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:44:01.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:44:01.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:44:01.106 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:44:01.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:44:01.534 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:44:01.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:44:01.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:44:02.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:44:02.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:44:02.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:44:02.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-04-12T16:44:02.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:44:02.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-04-12T16:44:02.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-04-12T16:44:02.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:44:02.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-04-12T16:44:02.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:464: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-04-12T16:44:02.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:44:02.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:44:02.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-04-12T16:44:02.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:44:02.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:44:02.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:44:02.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:44:02.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:44:02.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-04-12T16:44:02.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:466: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-04-12T16:44:03.586 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already out. 2024-04-12T16:44:03.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:469: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-04-12T16:44:03.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:44:03.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:44:03.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:44:03.603 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:44:03.604 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:44:03.604 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:44:03.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:44:03.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:44:03.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:44:03.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:44:03.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:44:03.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:44:03.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:44:03.711 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:44:03.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:44:04.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:44:04.741 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:44:04.741 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:44:04.741 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:44:04.741 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:44:04.742 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:44:04.742 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:44:04.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:44:04.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:44:04.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:44:05.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803838 2024-04-12T16:44:05.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803838 2024-04-12T16:44:05.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838' 2024-04-12T16:44:05.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:44:05.055 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:44:05.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574908 2024-04-12T16:44:05.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574908 2024-04-12T16:44:05.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574908' 2024-04-12T16:44:05.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:44:05.486 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:44:05.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378677 2024-04-12T16:44:05.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378677 2024-04-12T16:44:05.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574908 2-81604378677' 2024-04-12T16:44:05.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:44:05.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:44:06.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084324 2024-04-12T16:44:06.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084324 2024-04-12T16:44:06.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574908 2-81604378677 3-120259084324' 2024-04-12T16:44:06.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:44:06.004 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:44:06.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659163 2024-04-12T16:44:06.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659163 2024-04-12T16:44:06.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574908 2-81604378677 3-120259084324 4-176093659163' 2024-04-12T16:44:06.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:44:06.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:44:06.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332123 2024-04-12T16:44:06.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332123 2024-04-12T16:44:06.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574908 2-81604378677 3-120259084324 4-176093659163 5-219043332123' 2024-04-12T16:44:06.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:44:06.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:44:06.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037777 2024-04-12T16:44:06.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037777 2024-04-12T16:44:06.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574908 2-81604378677 3-120259084324 4-176093659163 5-219043332123 6-257698037777' 2024-04-12T16:44:06.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:44:06.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803838 2024-04-12T16:44:06.462 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:44:06.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:44:06.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803838 2024-04-12T16:44:06.464 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:44:06.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803838 2024-04-12T16:44:06.465 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803838 2024-04-12T16:44:06.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-04-12T16:44:06.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:44:06.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803839 -lt 25769803838 2024-04-12T16:44:06.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:44:06.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574908 2024-04-12T16:44:06.974 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:44:06.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:44:06.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574908 2024-04-12T16:44:06.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:44:06.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574908 2024-04-12T16:44:06.978 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574908 2024-04-12T16:44:06.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574908' 2024-04-12T16:44:06.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:44:07.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574909 -lt 55834574908 2024-04-12T16:44:07.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:44:07.442 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378677 2024-04-12T16:44:07.442 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:44:07.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:44:07.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378677 2024-04-12T16:44:07.445 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:44:07.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378677 2024-04-12T16:44:07.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378677' 2024-04-12T16:44:07.448 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378677 2024-04-12T16:44:07.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:44:07.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378677 -lt 81604378677 2024-04-12T16:44:07.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:44:07.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084324 2024-04-12T16:44:07.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:44:07.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:44:07.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084324 2024-04-12T16:44:07.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:44:07.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084324 2024-04-12T16:44:07.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084324' 2024-04-12T16:44:07.863 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 120259084324 2024-04-12T16:44:07.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:44:08.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084324 -lt 120259084324 2024-04-12T16:44:08.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:44:08.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659163 2024-04-12T16:44:08.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:44:08.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:44:08.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659163 2024-04-12T16:44:08.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:44:08.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659163 2024-04-12T16:44:08.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659163' 2024-04-12T16:44:08.350 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 176093659163 2024-04-12T16:44:08.350 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:44:08.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659164 -lt 176093659163 2024-04-12T16:44:08.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:44:08.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332123 2024-04-12T16:44:08.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:44:08.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:44:08.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332123 2024-04-12T16:44:08.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:44:08.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332123 2024-04-12T16:44:08.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332123' 2024-04-12T16:44:08.743 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 219043332123 2024-04-12T16:44:08.743 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:44:09.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332124 -lt 219043332123 2024-04-12T16:44:09.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:44:09.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-257698037777 2024-04-12T16:44:09.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:44:09.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:44:09.171 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:44:09.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-257698037777 2024-04-12T16:44:09.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037777 2024-04-12T16:44:09.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 257698037777' 2024-04-12T16:44:09.173 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 257698037777 2024-04-12T16:44:09.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:44:09.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037778 -lt 257698037777 2024-04-12T16:44:09.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:44:09.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:44:09.560 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:44:10.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:44:10.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:44:10.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:44:10.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:44:10.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:44:10.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:44:10.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:44:10.134 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:44:10.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T16:44:10.604 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:44:10.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:44:10.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:44:11.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-12T16:44:11.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-12T16:44:11.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-12T16:44:11.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-12T16:44:11.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-12T16:44:11.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-12T16:44:11.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-12T16:44:11.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:44:11.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:44:11.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:44:11.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:44:11.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:44:11.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:44:11.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:44:11.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:44:11.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:44:11.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:44:11.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:44:12.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:44:12.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:44:12.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:44:12.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-04-12T16:44:12.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T16:44:12.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T16:44:12.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-04-12T16:44:12.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T16:44:12.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T16:44:12.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-04-12T16:44:12.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T16:44:12.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T16:44:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-04-12T16:44:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T16:44:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T16:44:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-04-12T16:44:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T16:44:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T16:44:12.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-04-12T16:44:12.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T16:44:12.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T16:44:12.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:473: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-04-12T16:44:12.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T16:44:12.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T16:44:12.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-04-12T16:44:12.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T16:44:12.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T16:44:12.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-04-12T16:44:12.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T16:44:12.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T16:44:12.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-04-12T16:44:12.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:44:12.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T16:44:13.019 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T16:44:13.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-04-12T16:44:13.815 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile myprofile does not exist 2024-04-12T16:44:13.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-04-12T16:44:13.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:44:13.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:44:13.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:44:13.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:44:13.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:44:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:44:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:44:13.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:44:14.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:44:14.150 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:44:14.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:44:14.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:44:14.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:44:14.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:44:14.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:44:14.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:44:14.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:44:14.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:44:14.155 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:44:14.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:44:14.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:44:14.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:44:14.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:44:14.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:14.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:14.200 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:44:14.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:44:14.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:44:14.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:44:14.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:44:14.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:44:14.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:44:14.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:44:14.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:44:14.206 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:44:14.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:44:14.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:44:14.208 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:44:14.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:44:14.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:44:14.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:44:14.209 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:44:14.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:44:14.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:44:14.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:44:14.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:44:14.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:14.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:14.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:44:14.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:44:14.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:44:14.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T16:44:14.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:44:14.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:14.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:14.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T16:44:14.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:44:14.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T16:44:14.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T16:44:14.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:44:14.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:44:14.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:44:14.752 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:44:14.752 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:14.752 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:14.752 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:44:14.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:44:14.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:44:14.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:44:14.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:44:14.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:44:14.997 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:44:14.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T16:44:15.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:44:15.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:44:15.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:44:15.114 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:44:15.115 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:44:15.116 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:44:15.116 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:44:15.116 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:44:15.116 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:44:15.116 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:15.116 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:15.116 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:44:15.116 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:44:15.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T16:44:15.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T16:44:15.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T16:44:15.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:44:15.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:44:15.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:44:15.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T16:44:15.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:44:16.204 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:44:16.204 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:44:16.204 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:44:16.204 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:44:16.204 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:16.204 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:16.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:44:16.205 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:44:16.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:44:16.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T16:44:16.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:44:16.548 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:44:16.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:44:17.569 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T16:44:17.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:44:17.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:44:17.570 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:44:17.571 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:17.571 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:17.571 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:44:17.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T16:44:17.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T16:44:17.670 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:44:17.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T16:44:17.685 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:44:14.797+0000 7f9ddae82c80 0 load: jerasure load: lrc 2024-04-12T16:44:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-04-12T16:44:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-04-12T16:44:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:481: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-04-12T16:44:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:44:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:44:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-04-12T16:44:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-04-12T16:44:17.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T16:44:17.687 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-04-12T16:44:17.687 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-04-12T16:44:17.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:44:17.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T16:44:17.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:44:17.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11b7c9d8-7db6-461e-9287-e1f031d7416b 2024-04-12T16:44:17.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 11b7c9d8-7db6-461e-9287-e1f031d7416b' 2024-04-12T16:44:17.696 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 11b7c9d8-7db6-461e-9287-e1f031d7416b 2024-04-12T16:44:17.696 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:44:17.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhZBlmrKzlKhAAIwVSViwWy7M3udjXTALi0A== 2024-04-12T16:44:17.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhZBlmrKzlKhAAIwVSViwWy7M3udjXTALi0A=="}' 2024-04-12T16:44:17.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11b7c9d8-7db6-461e-9287-e1f031d7416b -i td/test-erasure-eio/0/new.json 2024-04-12T16:44:18.092 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:44:18.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T16:44:18.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhZBlmrKzlKhAAIwVSViwWy7M3udjXTALi0A== --osd-uuid 11b7c9d8-7db6-461e-9287-e1f031d7416b 2024-04-12T16:44:18.144 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:18.145+0000 7f7055abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:44:18.177 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:18.181+0000 7f7055abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:44:18.178 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:18.181+0000 7f7055abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:44:18.179 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:18.181+0000 7f7055abf7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:44:18.179 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:18.181+0000 7f7055abf7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:44:18.179 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:18.181+0000 7f7055abf7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:44:18.217 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:18.221+0000 7f7055abf7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T16:44:26.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T16:44:26.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:44:26.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:44:26.111 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:44:26.112 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:44:26.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:44:26.702 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:44:26.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:44:26.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:44:26.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:44:26.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:44:26.732 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:26.733+0000 7f949d3747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:44:26.734 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:26.737+0000 7f949d3747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:44:26.735 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:26.737+0000 7f949d3747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:44:27.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:44:27.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:44:27.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:44:27.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:44:27.117 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:44:27.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:44:27.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:27.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:44:27.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:27.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:27.290 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:27.293+0000 7f949d3747c0 -1 Falling back to public interface 2024-04-12T16:44:27.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:28.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:28.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:28.543 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:44:28.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:44:28.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:28.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:28.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:29.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:29.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:29.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:44:29.962 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:44:29.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:29.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:30.118 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:44:30.121+0000 7f949d3747c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:44:30.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:31.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:31.382 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:44:31.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:44:31.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:31.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:31.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:32.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:32.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:32.823 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:44:32.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:44:32.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:32.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:33.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:34.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:34.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:34.400 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:44:34.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:44:34.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:34.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:34.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:35.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:35.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:35.912 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:44:35.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:44:35.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:35.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:36.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:37.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:37.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:37.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:44:37.519 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:44:37.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:37.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:38.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:39.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:39.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:39.178 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:44:39.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:44:39.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:39.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:39.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:40.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:40.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:40.626 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:44:40.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:44:40.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:40.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:41.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:42.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:42.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:42.453 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:44:42.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:44:42.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:42.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:42.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:43.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:43.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:43.885 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:44:43.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:44:43.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:43.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:44.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:45.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:45.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:45.491 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:44:45.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:44:45.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:45.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:45.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:46.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:46.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:46.950 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:44:46.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:44:46.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:46.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:47.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:48.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:48.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:48.665 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:44:48.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:44:48.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:48.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:49.432 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:50.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:50.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:50.434 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:44:50.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:44:50.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:50.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:51.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:52.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:52.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:52.374 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:44:52.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:44:52.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:52.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:53.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:54.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:54.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:54.168 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:44:54.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:44:54.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:54.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:55.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:56.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:56.100 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:44:56.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:56.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:44:56.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:56.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:56.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:57.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:57.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:57.912 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:44:57.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:44:57.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:57.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:44:58.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:44:59.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:44:59.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:44:59.520 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:44:59.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:44:59.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:44:59.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:45:00.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:01.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:01.415 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:45:01.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:45:01.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:01.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:45:02.046 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2092490584,v1:127.0.0.1:6803/2092490584] [v2:127.0.0.1:6804/2092490584,v1:127.0.0.1:6805/2092490584] exists,up 11b7c9d8-7db6-461e-9287-e1f031d7416b 2024-04-12T16:45:02.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:45:02.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:45:02.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:45:02.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:45:02.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T16:45:02.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:45:02.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:45:02.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:45:02.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:45:02.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:45:02.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:45:02.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T16:45:02.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:45:02.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0cefdbc8-16ae-4b6b-81a7-ccf9cda32875 2024-04-12T16:45:02.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0cefdbc8-16ae-4b6b-81a7-ccf9cda32875' 2024-04-12T16:45:02.052 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 0cefdbc8-16ae-4b6b-81a7-ccf9cda32875 2024-04-12T16:45:02.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:45:02.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOZRlmkNT9BBAAEIe/yD4ZjKK6OPkU5Tg1xQ== 2024-04-12T16:45:02.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOZRlmkNT9BBAAEIe/yD4ZjKK6OPkU5Tg1xQ=="}' 2024-04-12T16:45:02.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0cefdbc8-16ae-4b6b-81a7-ccf9cda32875 -i td/test-erasure-eio/1/new.json 2024-04-12T16:45:02.820 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:45:02.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T16:45:02.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOZRlmkNT9BBAAEIe/yD4ZjKK6OPkU5Tg1xQ== --osd-uuid 0cefdbc8-16ae-4b6b-81a7-ccf9cda32875 2024-04-12T16:45:02.870 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:02.873+0000 7f6570cda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:02.872 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:02.873+0000 7f6570cda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:02.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:02.877+0000 7f6570cda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:02.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:02.877+0000 7f6570cda7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:45:02.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:02.877+0000 7f6570cda7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:45:02.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:02.877+0000 7f6570cda7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:45:02.937 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:02.941+0000 7f6570cda7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T16:45:10.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T16:45:10.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:45:10.907 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:45:10.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:45:10.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:45:11.457 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:45:11.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:45:11.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:45:11.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:45:11.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:45:11.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:45:11.496 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:11.497+0000 7ff3a0e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:11.543 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:11.545+0000 7ff3a0e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:11.544 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:11.545+0000 7ff3a0e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:11.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:45:11.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:45:11.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:45:11.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:45:11.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:45:11.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:11.855 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:45:11.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:45:11.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:11.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:12.107 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:12.109+0000 7ff3a0e2b7c0 -1 Falling back to public interface 2024-04-12T16:45:12.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:13.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:13.317 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:45:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:45:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:13.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:13.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:14.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:14.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:14.731 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:45:14.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:45:14.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:14.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:14.878 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:14.881+0000 7ff3a0e2b7c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:45:15.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:16.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:16.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:16.363 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:45:16.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:45:16.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:16.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:16.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:17.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:17.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:17.859 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:45:17.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:45:17.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:17.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:18.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:19.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:19.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:19.593 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:45:19.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:45:19.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:19.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:20.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:21.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:21.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:21.128 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:45:21.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:45:21.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:21.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:21.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:22.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:22.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:22.788 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:45:22.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:45:22.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:22.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:23.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:24.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:24.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:24.435 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:45:24.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:45:24.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:24.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:25.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:26.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:26.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:26.014 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:45:26.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:45:26.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:26.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:26.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:27.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:27.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:27.494 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:45:27.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:45:27.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:27.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:28.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:29.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:29.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:29.119 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:45:29.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:45:29.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:29.120 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:29.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:30.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:30.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:30.659 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:45:30.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:45:30.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:30.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:31.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:32.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:32.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:32.342 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:45:32.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:45:32.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:32.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:32.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:33.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:33.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:33.940 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:45:33.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:45:33.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:33.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:34.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:35.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:35.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:35.772 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:45:35.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:45:35.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:35.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:36.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:37.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:37.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:37.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:45:37.954 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:45:37.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:37.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:38.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:39.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:39.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:39.776 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:45:39.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:45:39.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:39.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:40.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:41.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:41.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:41.784 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:45:41.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:45:41.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:41.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:43.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:44.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:44.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:44.188 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:45:44.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:45:44.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:44.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:45.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:46.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:46.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:46.302 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:45:46.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:45:46.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:46.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:45:46.867 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1655228563,v1:127.0.0.1:6811/1655228563] [v2:127.0.0.1:6812/1655228563,v1:127.0.0.1:6813/1655228563] exists,up 0cefdbc8-16ae-4b6b-81a7-ccf9cda32875 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:45:46.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:45:46.872 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:45:46.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:45:46.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:45:46.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:45:46.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:45:46.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:45:46.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:45:46.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:45:46.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:45:46.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:45:46.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=19d361dd-c080-4c04-80d0-788306f4fde3 2024-04-12T16:45:46.875 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 19d361dd-c080-4c04-80d0-788306f4fde3 2024-04-12T16:45:46.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 19d361dd-c080-4c04-80d0-788306f4fde3' 2024-04-12T16:45:46.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:45:46.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6ZRlmvZUGNhAAh5221tW1hKptXX0RkEeCAA== 2024-04-12T16:45:46.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6ZRlmvZUGNhAAh5221tW1hKptXX0RkEeCAA=="}' 2024-04-12T16:45:46.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 19d361dd-c080-4c04-80d0-788306f4fde3 -i td/test-erasure-eio/2/new.json 2024-04-12T16:45:48.098 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:45:48.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T16:45:48.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6ZRlmvZUGNhAAh5221tW1hKptXX0RkEeCAA== --osd-uuid 19d361dd-c080-4c04-80d0-788306f4fde3 2024-04-12T16:45:48.156 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:48.157+0000 7fb4704507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:48.194 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:48.197+0000 7fb4704507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:48.196 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:48.197+0000 7fb4704507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:48.196 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:48.197+0000 7fb4704507c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:45:48.196 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:48.197+0000 7fb4704507c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:45:48.196 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:48.197+0000 7fb4704507c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:45:48.240 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:48.241+0000 7fb4704507c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T16:45:56.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T16:45:56.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:45:56.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:45:56.276 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:45:56.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:45:56.866 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:45:56.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:45:56.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:45:56.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:45:56.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:45:56.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:45:56.903 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:56.905+0000 7f326002f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:56.955 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:56.957+0000 7f326002f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:56.957 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:56.961+0000 7f326002f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:45:57.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:45:57.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:45:57.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:45:57.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:45:57.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:45:57.287 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:45:57.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:57.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:45:57.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:45:57.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:57.531 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:45:57.533+0000 7f326002f7c0 -1 Falling back to public interface 2024-04-12T16:45:57.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:45:58.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:45:58.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:45:58.735 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:45:58.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:45:58.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:45:58.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:45:59.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:00.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:00.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:00.171 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:46:00.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:46:00.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:00.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:00.343 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:00.345+0000 7f326002f7c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:46:00.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:01.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:01.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:01.676 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:46:01.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:46:01.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:01.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:02.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:03.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:03.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:03.217 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:46:03.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:46:03.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:03.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:03.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:04.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:04.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:04.704 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:46:04.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:46:04.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:04.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:05.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:06.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:06.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:06.272 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:46:06.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:46:06.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:06.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:06.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:07.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:07.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:07.809 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:46:07.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:46:07.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:07.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:08.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:09.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:09.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:09.477 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:46:09.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:46:09.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:09.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:10.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:11.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:11.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:11.181 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:46:11.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:46:11.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:11.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:11.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:12.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:12.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:12.799 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:46:12.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:46:12.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:12.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:13.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:14.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:14.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:14.300 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:46:14.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:46:14.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:14.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:14.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:15.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:15.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:15.843 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:46:15.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:46:15.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:15.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:16.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:17.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:17.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:17.494 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:46:17.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:46:17.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:17.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:18.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:19.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:19.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:19.085 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:46:19.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:46:19.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:19.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:20.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:21.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:21.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:21.108 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:46:21.108 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:46:21.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:21.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:22.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:23.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:23.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:23.558 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:46:23.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:46:23.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:23.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:24.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:25.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:25.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:25.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:46:25.713 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:46:25.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:25.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:26.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:27.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:27.908 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:27.908 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:46:27.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:46:27.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:27.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:28.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:29.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:29.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:29.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:46:29.675 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:46:29.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:29.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:30.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:31.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:31.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:31.638 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:46:31.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:46:31.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:31.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:32.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:33.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:33.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:33.367 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:46:33.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:46:33.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:33.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:46:33.993 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3866778898,v1:127.0.0.1:6819/3866778898] [v2:127.0.0.1:6820/3866778898,v1:127.0.0.1:6821/3866778898] exists,up 19d361dd-c080-4c04-80d0-788306f4fde3 2024-04-12T16:46:33.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:46:33.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:46:33.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:46:33.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T16:46:34.000 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:46:34.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9b7ad502-fa96-4735-a012-37cac54dcb18 2024-04-12T16:46:34.001 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 9b7ad502-fa96-4735-a012-37cac54dcb18 2024-04-12T16:46:34.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9b7ad502-fa96-4735-a012-37cac54dcb18' 2024-04-12T16:46:34.001 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:46:34.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqZRlmSCrEARAAXawuD/fqUEB7ZGdSXIhUGg== 2024-04-12T16:46:34.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqZRlmSCrEARAAXawuD/fqUEB7ZGdSXIhUGg=="}' 2024-04-12T16:46:34.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9b7ad502-fa96-4735-a012-37cac54dcb18 -i td/test-erasure-eio/3/new.json 2024-04-12T16:46:35.724 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:46:35.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T16:46:35.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqZRlmSCrEARAAXawuD/fqUEB7ZGdSXIhUGg== --osd-uuid 9b7ad502-fa96-4735-a012-37cac54dcb18 2024-04-12T16:46:35.781 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:35.781+0000 7f59ccf627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:46:35.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:35.877+0000 7f59ccf627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:46:35.875 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:35.877+0000 7f59ccf627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:46:35.876 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:35.877+0000 7f59ccf627c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:46:35.876 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:35.877+0000 7f59ccf627c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:46:35.876 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:35.877+0000 7f59ccf627c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:46:36.073 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:36.077+0000 7f59ccf627c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T16:46:43.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T16:46:43.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:46:43.744 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T16:46:43.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T16:46:43.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:46:44.325 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:46:44.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T16:46:44.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:46:44.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:46:44.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:46:44.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:46:44.355 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:44.357+0000 7f7e386d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:46:44.357 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:44.361+0000 7f7e386d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:46:44.358 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:44.361+0000 7f7e386d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:46:44.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T16:46:44.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:46:44.733 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:46:44.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:46:44.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:46:44.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:46:44.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:44.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:46:44.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:44.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:44.902 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:44.905+0000 7f7e386d57c0 -1 Falling back to public interface 2024-04-12T16:46:45.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:46.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:46.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:46.139 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:46:46.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:46:46.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:46.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:46.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:47.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:47.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:47.578 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:46:47.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:46:47.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:47.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:47.882 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:46:47.885+0000 7f7e386d57c0 -1 osd.3 0 log_to_monitors true 2024-04-12T16:46:48.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:49.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:49.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:49.003 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:46:49.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:46:49.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:49.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:49.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:50.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:50.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:50.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:46:50.842 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:46:50.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:50.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:51.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:52.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:52.683 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:46:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:46:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:52.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:53.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:54.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:54.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:54.223 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:46:54.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:46:54.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:54.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:54.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:55.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:55.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:55.695 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:46:55.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:46:55.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:55.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:56.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:57.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:57.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:57.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:46:57.282 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:46:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:57.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:57.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:46:58.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:46:58.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:46:58.760 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:46:58.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:46:58.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:46:58.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:46:59.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:00.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:00.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:00.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:47:00.275 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:47:00.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:00.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:00.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:01.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:01.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:01.772 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:47:01.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:47:01.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:01.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:02.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:03.368 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:47:03.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:47:03.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:03.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:03.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:04.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:04.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:04.790 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:47:04.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:47:04.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:04.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:05.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:06.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:06.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:06.426 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:47:06.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:47:06.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:06.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:07.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:08.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:08.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:08.036 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:47:08.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:47:08.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:08.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:08.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:09.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:09.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:09.875 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:47:09.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:47:09.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:09.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:10.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:11.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:11.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:11.868 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:47:11.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:47:11.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:11.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:12.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:13.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:13.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:13.635 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:47:13.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:47:13.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:13.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:14.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:15.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:15.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:15.873 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:47:15.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:47:15.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:15.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:16.743 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:17.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:17.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:17.745 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:47:17.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:47:17.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:17.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:18.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:19.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:19.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:19.768 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:47:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:47:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:19.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:47:20.308 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2984469876,v1:127.0.0.1:6827/2984469876] [v2:127.0.0.1:6828/2984469876,v1:127.0.0.1:6829/2984469876] exists,up 9b7ad502-fa96-4735-a012-37cac54dcb18 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-12T16:47:20.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:47:20.311 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:47:20.312 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:47:20.312 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:47:20.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:47:20.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-04-12T16:47:20.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:47:20.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bce42655-55d3-4090-88f2-b30e6f42ff7d 2024-04-12T16:47:20.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 bce42655-55d3-4090-88f2-b30e6f42ff7d' 2024-04-12T16:47:20.315 INFO:tasks.workunit.client.0.smithi137.stdout:add osd4 bce42655-55d3-4090-88f2-b30e6f42ff7d 2024-04-12T16:47:20.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:47:20.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYZRlmg095FBAAY37SWdq6KHWfKk3ljyVRXw== 2024-04-12T16:47:20.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYZRlmg095FBAAY37SWdq6KHWfKk3ljyVRXw=="}' 2024-04-12T16:47:20.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bce42655-55d3-4090-88f2-b30e6f42ff7d -i td/test-erasure-eio/4/new.json 2024-04-12T16:47:21.823 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:47:21.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-04-12T16:47:21.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYZRlmg095FBAAY37SWdq6KHWfKk3ljyVRXw== --osd-uuid bce42655-55d3-4090-88f2-b30e6f42ff7d 2024-04-12T16:47:21.879 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:21.881+0000 7fcb758fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:47:21.921 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:21.925+0000 7fcb758fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:47:21.923 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:21.925+0000 7fcb758fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:47:21.924 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:21.925+0000 7fcb758fb7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:47:21.924 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:21.925+0000 7fcb758fb7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:47:21.924 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:21.925+0000 7fcb758fb7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:47:22.224 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:22.229+0000 7fcb758fb7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-04-12T16:47:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-04-12T16:47:30.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:47:30.464 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd4 key to auth repository 2024-04-12T16:47:30.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-12T16:47:30.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:47:31.083 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.4 2024-04-12T16:47:31.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-12T16:47:31.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:47:31.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:47:31.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:47:31.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:47:31.121 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:31.125+0000 7f75376e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:47:31.149 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:31.153+0000 7f75376e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:47:31.151 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:31.153+0000 7f75376e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:47:31.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-12T16:47:31.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:47:31.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-12T16:47:31.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:47:31.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:47:31.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:31.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:47:31.541 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:47:31.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:31.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:31.735 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:31.737+0000 7f75376e27c0 -1 Falling back to public interface 2024-04-12T16:47:32.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:33.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:33.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:33.015 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:47:33.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:47:33.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:33.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:33.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:34.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:34.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:34.474 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:47:34.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:47:34.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:34.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:34.508 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:47:34.509+0000 7f75376e27c0 -1 osd.4 0 log_to_monitors true 2024-04-12T16:47:34.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:35.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:35.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:35.962 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:47:35.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:47:35.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:35.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:36.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:37.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:37.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:37.503 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:47:37.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:47:37.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:37.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:38.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:39.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:39.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:39.670 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:47:39.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:47:39.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:39.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:40.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:41.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:41.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:41.239 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:47:41.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:47:41.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:41.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:41.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:42.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:42.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:42.873 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:47:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:47:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:42.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:43.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:44.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:44.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:44.389 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:47:44.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:47:44.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:44.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:44.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:45.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:45.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:45.977 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:47:45.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:47:45.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:45.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:46.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:47.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:47.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:47.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:47:47.682 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:47:47.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:47.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:48.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:49.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:49.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:49.154 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:47:49.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:47:49.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:49.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:49.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:50.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:50.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:50.728 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:47:50.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:47:50.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:50.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:51.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:52.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:52.212 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:47:52.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:47:52.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:52.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:52.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:53.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:53.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:53.916 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:47:53.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:47:53.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:53.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:54.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:55.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:55.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:55.460 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:47:55.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:47:55.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:55.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:56.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:57.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:57.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:57.483 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:47:57.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:47:57.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:57.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:47:58.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:47:59.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:47:59.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:47:59.931 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:47:59.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:47:59.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:47:59.932 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:48:00.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:01.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:01.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:01.989 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:48:01.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:48:01.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:01.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:48:02.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:03.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:03.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:03.822 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:48:03.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:48:03.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:03.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:48:04.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:05.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:05.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:05.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:48:05.611 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:48:05.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:05.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:48:06.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:07.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:07.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:07.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:48:07.821 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:48:07.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:07.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:48:08.285 INFO:tasks.workunit.client.0.smithi137.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1366710768,v1:127.0.0.1:6835/1366710768] [v2:127.0.0.1:6836/1366710768,v1:127.0.0.1:6837/1366710768] exists,up bce42655-55d3-4090-88f2-b30e6f42ff7d 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:48:08.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:48:08.290 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:48:08.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:48:08.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-04-12T16:48:08.292 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:48:08.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=364f0db3-90fa-4127-bc24-da35b3e1ddf8 2024-04-12T16:48:08.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 364f0db3-90fa-4127-bc24-da35b3e1ddf8' 2024-04-12T16:48:08.293 INFO:tasks.workunit.client.0.smithi137.stdout:add osd5 364f0db3-90fa-4127-bc24-da35b3e1ddf8 2024-04-12T16:48:08.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:48:08.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIZRlmcPdJExAAcn43Rv5GAV9vwLt2LDpB+w== 2024-04-12T16:48:08.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIZRlmcPdJExAAcn43Rv5GAV9vwLt2LDpB+w=="}' 2024-04-12T16:48:08.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 364f0db3-90fa-4127-bc24-da35b3e1ddf8 -i td/test-erasure-eio/5/new.json 2024-04-12T16:48:09.569 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:48:09.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-04-12T16:48:09.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIZRlmcPdJExAAcn43Rv5GAV9vwLt2LDpB+w== --osd-uuid 364f0db3-90fa-4127-bc24-da35b3e1ddf8 2024-04-12T16:48:09.636 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:09.637+0000 7f94269937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:09.638 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:09.641+0000 7f94269937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:09.639 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:09.641+0000 7f94269937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:09.640 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:09.641+0000 7f94269937c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:48:09.640 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:09.641+0000 7f94269937c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:48:09.640 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:09.641+0000 7f94269937c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:48:09.894 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:09.897+0000 7f94269937c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-04-12T16:48:19.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-04-12T16:48:19.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:48:19.287 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd5 key to auth repository 2024-04-12T16:48:19.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-12T16:48:19.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:48:19.915 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.5 2024-04-12T16:48:19.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-12T16:48:19.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:48:19.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:48:19.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:48:19.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:48:19.952 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:19.952+0000 7f25aa7207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:19.996 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:19.996+0000 7f25aa7207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:19.998 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:20.000+0000 7f25aa7207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:20.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-12T16:48:20.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:48:20.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-12T16:48:20.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:48:20.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:48:20.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:20.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:48:20.340 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:48:20.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:20.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:20.567 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:20.568+0000 7f25aa7207c0 -1 Falling back to public interface 2024-04-12T16:48:20.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:21.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:21.807 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:48:21.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:21.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:48:21.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:21.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:22.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:23.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:23.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:23.223 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:48:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:48:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:23.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:23.586 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:23.588+0000 7f25aa7207c0 -1 osd.5 0 log_to_monitors true 2024-04-12T16:48:23.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:24.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:24.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:24.694 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:48:24.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:48:24.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:24.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:25.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:26.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:26.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:26.763 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:48:26.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:48:26.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:26.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:27.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:28.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:28.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:28.424 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:48:28.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:48:28.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:28.425 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:28.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:29.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:29.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:29.994 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:48:29.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:48:29.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:29.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:30.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:31.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:31.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:31.461 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:48:31.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:48:31.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:31.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:32.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:33.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:33.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:33.141 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:48:33.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:48:33.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:33.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:33.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:34.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:34.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:34.674 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:48:34.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:48:34.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:34.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:35.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:36.126 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:36.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:36.127 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:48:36.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:48:36.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:36.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:36.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:37.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:37.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:37.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:48:37.607 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:48:37.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:37.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:38.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:39.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:39.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:39.219 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:48:39.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:48:39.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:39.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:39.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:40.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:40.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:40.772 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:48:40.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:48:40.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:40.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:41.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:42.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:42.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:42.238 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:48:42.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:48:42.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:42.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:42.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:43.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:43.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:43.748 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:48:43.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:48:43.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:43.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:44.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:45.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:45.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:45.637 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:48:45.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:48:45.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:45.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:47.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:48.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:48.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:48.056 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:48:48.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:48:48.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:48.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:49.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:50.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:50.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:50.344 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:48:50.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:48:50.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:50.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:51.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:52.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:52.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:52.478 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:48:52.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:48:52.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:52.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:53.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:54.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:54.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:54.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:48:54.196 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:48:54.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:54.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:55.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:56.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:56.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:56.021 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:48:56.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:48:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:56.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:56.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:48:57.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:48:57.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:48:57.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:48:57.519 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:48:57.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:48:57.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:48:58.212 INFO:tasks.workunit.client.0.smithi137.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2864213089,v1:127.0.0.1:6843/2864213089] [v2:127.0.0.1:6844/2864213089,v1:127.0.0.1:6845/2864213089] exists,up 364f0db3-90fa-4127-bc24-da35b3e1ddf8 2024-04-12T16:48:58.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:48:58.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:48:58.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:48:58.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:48:58.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:48:58.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:48:58.220 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:48:58.220 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:48:58.220 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:48:58.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:48:58.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:48:58.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:48:58.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:48:58.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:48:58.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-04-12T16:48:58.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:48:58.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6800a0f9-324d-41eb-9c45-51436b5803d5 2024-04-12T16:48:58.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 6800a0f9-324d-41eb-9c45-51436b5803d5' 2024-04-12T16:48:58.223 INFO:tasks.workunit.client.0.smithi137.stdout:add osd6 6800a0f9-324d-41eb-9c45-51436b5803d5 2024-04-12T16:48:58.224 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:48:58.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6ZRlmZe3YDhAAEk/m/mI20wZDJyXy+AcFAQ== 2024-04-12T16:48:58.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6ZRlmZe3YDhAAEk/m/mI20wZDJyXy+AcFAQ=="}' 2024-04-12T16:48:58.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6800a0f9-324d-41eb-9c45-51436b5803d5 -i td/test-erasure-eio/6/new.json 2024-04-12T16:48:59.871 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:48:59.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-04-12T16:48:59.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6ZRlmZe3YDhAAEk/m/mI20wZDJyXy+AcFAQ== --osd-uuid 6800a0f9-324d-41eb-9c45-51436b5803d5 2024-04-12T16:48:59.938 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:59.940+0000 7f8bf57547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:59.980 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:59.984+0000 7f8bf57547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:59.981 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:59.984+0000 7f8bf57547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:48:59.982 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:59.984+0000 7f8bf57547c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:48:59.983 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:59.984+0000 7f8bf57547c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:48:59.983 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:48:59.984+0000 7f8bf57547c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T16:49:00.184 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:49:00.184+0000 7f8bf57547c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-04-12T16:49:08.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-04-12T16:49:08.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:49:08.478 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd6 key to auth repository 2024-04-12T16:49:08.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-12T16:49:08.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:49:09.102 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.6 2024-04-12T16:49:09.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-12T16:49:09.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:49:09.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:49:09.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:49:09.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:49:09.141 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:49:09.140+0000 7f05ac5a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:49:09.167 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:49:09.168+0000 7f05ac5a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:49:09.168 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:49:09.168+0000 7f05ac5a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:49:09.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-12T16:49:09.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:49:09.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-12T16:49:09.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:49:09.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:49:09.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:09.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:49:09.576 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:49:09.576 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:09.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:09.723 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:49:09.724+0000 7f05ac5a47c0 -1 Falling back to public interface 2024-04-12T16:49:10.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:11.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:11.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:11.017 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:49:11.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:49:11.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:11.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:11.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:12.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:12.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:12.492 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:49:12.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:49:12.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:12.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:12.555 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:49:12.556+0000 7f05ac5a47c0 -1 osd.6 0 log_to_monitors true 2024-04-12T16:49:13.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:14.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:14.092 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:14.092 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:49:14.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:49:14.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:14.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:15.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:16.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:16.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:16.133 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:49:16.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:49:16.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:16.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:16.965 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:17.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:17.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:17.967 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:49:17.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:49:17.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:17.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:18.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:19.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:19.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:19.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:49:19.383 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:49:19.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:19.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:19.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:20.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:20.892 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:20.893 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:49:20.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:49:20.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:20.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:21.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:22.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:22.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:22.387 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:49:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:49:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:22.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:24.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:24.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:24.010 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:49:24.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:49:24.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:24.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:24.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:25.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:25.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:25.649 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:49:25.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:49:25.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:25.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:26.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:27.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:27.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:27.222 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:49:27.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:49:27.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:27.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:27.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:28.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:28.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:28.846 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:49:28.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:49:28.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:28.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:29.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:30.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:30.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:30.361 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:49:30.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:49:30.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:30.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:30.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:31.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:31.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:31.902 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:49:31.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:49:31.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:31.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:32.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:33.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:33.652 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:33.652 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:49:33.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:49:33.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:33.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:34.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:35.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:35.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:35.107 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:49:35.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:49:35.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:35.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:35.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:36.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:36.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:36.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:49:36.995 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:49:36.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:36.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:37.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:38.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:38.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:38.677 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:49:38.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:49:38.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:38.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:39.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:40.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:40.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:40.505 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:49:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:49:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:41.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:42.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:42.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:42.514 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:49:42.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:49:42.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:42.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:43.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:44.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:44.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:44.243 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:49:44.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:49:44.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:44.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:45.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:49:46.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:49:46.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:49:46.180 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:49:46.181 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:49:46.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:49:46.181 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T16:49:46.619 INFO:tasks.workunit.client.0.smithi137.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3699814380,v1:127.0.0.1:6851/3699814380] [v2:127.0.0.1:6852/3699814380,v1:127.0.0.1:6853/3699814380] exists,up 6800a0f9-324d-41eb-9c45-51436b5803d5 2024-04-12T16:49:46.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:49:46.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:49:46.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:49:46.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T16:49:46.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:49:46.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:49:46.621 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:49:46.621 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:49:46.621 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:49:46.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T16:49:46.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T16:49:46.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T16:49:46.734 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:49:46.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T16:49:46.754 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:44:27.297+0000 7f949d3747c0 0 load: jerasure load: lrc 2024-04-12T16:49:46.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T16:49:46.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-04-12T16:49:47.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T16:49:47.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T16:49:49.519 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T16:49:49.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:49:50.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T16:49:50.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:49:50.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:49:50.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:49:50.545 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:49:50.545 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:49:50.545 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:49:50.546 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:49:50.546 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:49:50.546 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:49:50.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:49:50.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:49:50.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:49:50.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:49:50.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:49:50.674 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:49:51.201 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:49:51.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:49:51.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803838 2024-04-12T16:49:51.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803838 2024-04-12T16:49:51.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838' 2024-04-12T16:49:51.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:49:51.362 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:49:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542202 2024-04-12T16:49:51.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542202 2024-04-12T16:49:51.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542202' 2024-04-12T16:49:51.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:49:51.517 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:49:51.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345965 2024-04-12T16:49:51.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345965 2024-04-12T16:49:51.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542202 2-85899345965' 2024-04-12T16:49:51.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:49:51.698 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:49:51.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018919 2024-04-12T16:49:51.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018919 2024-04-12T16:49:51.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542202 2-85899345965 3-128849018919' 2024-04-12T16:49:51.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:49:51.872 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:49:52.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659164 2024-04-12T16:49:52.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659164 2024-04-12T16:49:52.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542202 2-85899345965 3-128849018919 4-176093659164' 2024-04-12T16:49:52.025 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:49:52.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:49:52.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332116 2024-04-12T16:49:52.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332116 2024-04-12T16:49:52.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542202 2-85899345965 3-128849018919 4-176093659164 5-219043332116' 2024-04-12T16:49:52.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:49:52.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:49:52.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005067 2024-04-12T16:49:52.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005067 2024-04-12T16:49:52.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542202 2-85899345965 3-128849018919 4-176093659164 5-219043332116 6-261993005067' 2024-04-12T16:49:52.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:49:52.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803838 2024-04-12T16:49:52.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:49:52.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:49:52.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803838 2024-04-12T16:49:52.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:49:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803838 2024-04-12T16:49:52.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-04-12T16:49:52.345 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803838 2024-04-12T16:49:52.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:49:52.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803836 -lt 25769803838 2024-04-12T16:49:52.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T16:49:53.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T16:49:53.824 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:49:54.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803839 -lt 25769803838 2024-04-12T16:49:54.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:49:54.241 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-60129542202 2024-04-12T16:49:54.242 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:49:54.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:49:54.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-60129542202 2024-04-12T16:49:54.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:49:54.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542202 2024-04-12T16:49:54.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 60129542202' 2024-04-12T16:49:54.246 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 60129542202 2024-04-12T16:49:54.247 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:49:54.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542203 -lt 60129542202 2024-04-12T16:49:54.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:49:54.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345965 2024-04-12T16:49:54.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:49:54.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:49:54.634 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345965 2024-04-12T16:49:54.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:49:54.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345965 2024-04-12T16:49:54.636 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 85899345965 2024-04-12T16:49:54.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345965' 2024-04-12T16:49:54.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:49:55.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345965 -lt 85899345965 2024-04-12T16:49:55.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:49:55.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018919 2024-04-12T16:49:55.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:49:55.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:49:55.089 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018919 2024-04-12T16:49:55.089 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:49:55.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018919 2024-04-12T16:49:55.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018919' 2024-04-12T16:49:55.091 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 128849018919 2024-04-12T16:49:55.092 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:49:55.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018920 -lt 128849018919 2024-04-12T16:49:55.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:49:55.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659164 2024-04-12T16:49:55.564 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:49:55.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:49:55.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659164 2024-04-12T16:49:55.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:49:55.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659164 2024-04-12T16:49:55.568 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 176093659164 2024-04-12T16:49:55.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659164' 2024-04-12T16:49:55.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:49:55.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659165 -lt 176093659164 2024-04-12T16:49:55.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:49:55.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332116 2024-04-12T16:49:55.967 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:49:55.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:49:55.970 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332116 2024-04-12T16:49:55.970 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:49:55.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332116 2024-04-12T16:49:55.971 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 219043332116 2024-04-12T16:49:55.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332116' 2024-04-12T16:49:55.971 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:49:56.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332117 -lt 219043332116 2024-04-12T16:49:56.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:49:56.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:49:56.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005067 2024-04-12T16:49:56.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:49:56.394 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005067 2024-04-12T16:49:56.395 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:49:56.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005067 2024-04-12T16:49:56.396 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 261993005067 2024-04-12T16:49:56.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005067' 2024-04-12T16:49:56.396 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:49:56.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005067 -lt 261993005067 2024-04-12T16:49:56.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:49:56.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:49:56.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:49:57.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:49:57.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:49:57.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:49:57.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:49:57.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:49:57.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:49:57.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:49:57.454 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:49:57.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:49:57.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:49:57.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:49:57.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:49:58.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:49:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:49:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:49:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-04-12T16:49:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:49:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:49:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-04-12T16:49:58.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:49:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:49:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:49:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:49:58.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:49:58.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-04-12T16:49:58.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-04-12T16:49:58.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:49:58.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:49:58.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:49:58.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-04-12T16:49:59.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-04-12T16:49:59.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:49:59.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:49:59.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T16:49:59.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:49:59.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:49:59.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:49:59.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-04-12T16:49:59.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:49:59.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:49:59.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:49:59.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-12T16:49:59.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:49:59.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-04-12T16:49:59.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:49:59.358 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T16:49:59.359 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:49:59.359 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:49:59.796 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:49:59.797 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-04-12T16:49:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:49:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-12T16:49:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:49:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-12T16:49:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:49:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:49:59.799 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:49:59.799 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-12T16:49:59.799 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T16:49:59.799 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T16:49:59.800 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:49:59.800 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:49:59.800 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:49:59.800 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T16:49:59.801 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:49:59.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:49:59.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:49:59.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:49:59.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-04-12T16:49:59.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T16:49:59.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T16:49:59.916 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:49:59.916 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:49:59.916 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:49:59.917 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T16:49:59.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:49:59.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-04-12T16:50:00.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:497: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-04-12T16:50:00.622 INFO:tasks.workunit.client.0.smithi137.stderr:osd.0 is already out. 2024-04-12T16:50:00.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-04-12T16:50:00.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:50:00.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:50:00.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:50:00.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:50:00.643 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:50:00.643 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:50:00.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:50:00.643 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:50:00.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:50:00.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:50:00.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:50:00.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:50:00.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:50:00.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:50:00.750 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:50:01.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:50:01.870 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:50:02.210 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803843 2024-04-12T16:50:02.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803843 2024-04-12T16:50:02.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843' 2024-04-12T16:50:02.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:50:02.211 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:50:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542206 2024-04-12T16:50:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542206 2024-04-12T16:50:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-60129542206' 2024-04-12T16:50:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:50:02.580 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:50:02.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345969 2024-04-12T16:50:02.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345969 2024-04-12T16:50:02.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-60129542206 2-85899345969' 2024-04-12T16:50:02.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:50:02.741 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:50:03.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018923 2024-04-12T16:50:03.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018923 2024-04-12T16:50:03.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-60129542206 2-85899345969 3-128849018923' 2024-04-12T16:50:03.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:50:03.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:50:03.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659168 2024-04-12T16:50:03.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659168 2024-04-12T16:50:03.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-60129542206 2-85899345969 3-128849018923 4-176093659168' 2024-04-12T16:50:03.196 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:50:03.196 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T16:50:03.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332120 2024-04-12T16:50:03.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332120 2024-04-12T16:50:03.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-60129542206 2-85899345969 3-128849018923 4-176093659168 5-219043332120' 2024-04-12T16:50:03.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:50:03.352 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T16:50:03.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005071 2024-04-12T16:50:03.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005071 2024-04-12T16:50:03.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-60129542206 2-85899345969 3-128849018923 4-176093659168 5-219043332120 6-261993005071' 2024-04-12T16:50:03.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:50:03.506 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803843 2024-04-12T16:50:03.506 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:50:03.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:50:03.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803843 2024-04-12T16:50:03.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:50:03.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803843 2024-04-12T16:50:03.508 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803843 2024-04-12T16:50:03.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803843' 2024-04-12T16:50:03.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:50:04.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803843 -lt 25769803843 2024-04-12T16:50:04.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:50:04.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:50:04.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-60129542206 2024-04-12T16:50:04.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:50:04.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-60129542206 2024-04-12T16:50:04.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:50:04.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542206 2024-04-12T16:50:04.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 60129542206' 2024-04-12T16:50:04.086 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 60129542206 2024-04-12T16:50:04.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:50:04.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542207 -lt 60129542206 2024-04-12T16:50:04.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:50:04.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345969 2024-04-12T16:50:04.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:50:04.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:50:04.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345969 2024-04-12T16:50:04.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:50:04.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345969 2024-04-12T16:50:04.529 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 85899345969 2024-04-12T16:50:04.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345969' 2024-04-12T16:50:04.529 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:50:04.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345969 -lt 85899345969 2024-04-12T16:50:04.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:50:04.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018923 2024-04-12T16:50:04.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:50:04.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:50:04.992 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018923 2024-04-12T16:50:04.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:50:04.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018923 2024-04-12T16:50:04.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018923' 2024-04-12T16:50:04.994 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 128849018923 2024-04-12T16:50:04.995 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:50:05.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018924 -lt 128849018923 2024-04-12T16:50:05.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:50:05.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659168 2024-04-12T16:50:05.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:50:05.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:50:05.394 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659168 2024-04-12T16:50:05.394 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:50:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659168 2024-04-12T16:50:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659168' 2024-04-12T16:50:05.397 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 176093659168 2024-04-12T16:50:05.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:50:05.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659169 -lt 176093659168 2024-04-12T16:50:05.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:50:05.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-219043332120 2024-04-12T16:50:05.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:50:05.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T16:50:05.839 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-219043332120 2024-04-12T16:50:05.839 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:50:05.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332120 2024-04-12T16:50:05.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 219043332120' 2024-04-12T16:50:05.841 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 219043332120 2024-04-12T16:50:05.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T16:50:06.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332121 -lt 219043332120 2024-04-12T16:50:06.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:50:06.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005071 2024-04-12T16:50:06.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:50:06.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T16:50:06.264 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:50:06.264 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005071 2024-04-12T16:50:06.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005071 2024-04-12T16:50:06.265 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 261993005071 2024-04-12T16:50:06.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005071' 2024-04-12T16:50:06.265 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T16:50:06.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005071 -lt 261993005071 2024-04-12T16:50:06.702 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:50:06.703 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:50:06.703 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:50:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:50:07.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:50:07.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:50:07.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:50:07.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:50:07.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:50:07.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:50:07.280 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:50:07.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T16:50:07.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:50:07.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:50:07.669 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:50:08.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-12T16:50:08.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-12T16:50:08.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-12T16:50:08.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-12T16:50:08.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-12T16:50:08.205 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-12T16:50:08.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-12T16:50:08.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:50:08.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:50:08.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:50:08.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:50:08.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:50:08.307 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:50:08.307 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:50:08.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T16:50:08.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:50:08.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:50:08.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:50:09.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-12T16:50:09.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-04-12T16:50:09.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-12T16:50:09.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-12T16:50:09.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-12T16:50:09.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-12T16:50:09.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-12T16:50:09.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-12T16:50:09.293 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-12T16:50:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1535 2024-04-12T16:50:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1535 '!=' null 2024-04-12T16:50:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-12T16:50:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-12T16:50:09.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-12T16:50:09.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-12T16:50:09.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:50:09.937 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:50:09.937 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:50:09.937 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:50:09.937 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:50:09.937 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:50:09.938 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:50:10.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:50:10.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:50:10.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:50:10.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:50:10.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:50:10.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:50:10.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:50:10.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-04-12T16:50:10.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T16:50:10.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T16:50:10.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-04-12T16:50:10.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T16:50:10.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T16:50:10.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-04-12T16:50:11.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T16:50:11.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T16:50:11.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-04-12T16:50:11.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T16:50:11.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T16:50:11.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-04-12T16:50:11.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T16:50:11.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T16:50:11.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-04-12T16:50:11.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T16:50:11.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T16:50:11.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-04-12T16:50:11.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T16:50:11.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T16:50:11.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-04-12T16:50:11.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T16:50:11.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T16:50:11.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-04-12T16:50:11.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T16:50:11.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T16:50:11.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-04-12T16:50:11.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:50:11.105 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T16:50:11.695 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T16:50:11.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-04-12T16:50:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile myprofile does not exist 2024-04-12T16:50:12.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-04-12T16:50:12.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:50:12.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:50:12.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:50:12.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:50:12.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:50:12.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:50:12.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:50:12.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:50:12.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:50:12.778 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:50:12.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:50:12.780 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:50:12.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:50:12.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:50:12.781 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:50:12.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:50:12.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:50:12.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:50:12.783 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:50:12.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:50:12.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:50:12.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:50:12.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:50:12.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:12.828 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:12.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:50:12.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:50:12.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:50:12.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T16:50:12.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T16:50:12.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T16:50:12.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T16:50:12.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:50:12.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:50:12.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:50:12.831 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:50:12.831 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:50:12.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:50:12.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:50:12.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:50:12.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:50:12.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:50:12.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:50:12.834 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:50:12.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:50:12.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:50:12.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:50:12.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:50:12.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:50:12.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:50:12.838 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:50:12.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:50:12.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:50:12.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:50:12.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:50:12.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:12.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:12.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:50:12.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:50:12.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:50:12.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T16:50:12.844 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:50:12.844 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:12.844 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:12.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T16:50:12.845 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:50:12.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:50:12.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:50:12.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T16:50:12.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T16:50:12.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T16:50:12.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:50:12.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:50:12.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:50:12.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T16:50:12.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T16:50:13.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:50:13.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:50:13.359 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:50:13.360 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:50:13.360 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:13.360 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:13.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:50:13.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:50:13.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:50:13.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:50:13.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:50:13.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:50:13.605 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:50:13.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:50:13.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:50:13.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:50:13.606 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:50:13.606 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:50:13.606 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:13.607 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:13.607 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:50:13.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:50:13.607 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T16:50:13.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:50:13.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:50:13.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:50:13.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:50:13.726 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:50:13.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T16:50:13.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T16:50:13.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T16:50:13.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:50:13.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:50:13.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:50:13.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T16:50:13.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:50:14.792 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:50:14.792 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:50:14.792 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:50:14.793 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:50:14.793 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:14.793 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:14.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:50:14.794 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:50:14.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:50:14.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T16:50:14.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:50:15.232 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:50:15.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:50:16.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T16:50:16.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:50:16.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:50:16.253 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:50:16.253 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:16.253 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:16.253 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:50:16.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T16:50:16.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T16:50:16.354 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:50:16.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T16:50:16.370 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:50:13.404+0000 7fdb2fe8bc80 0 load: jerasure load: lrc 2024-04-12T16:50:16.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-04-12T16:50:16.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-04-12T16:50:16.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: local objname=myobject 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: local lastobj=100 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_recovery_unfound: local testobj=obj75 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_recovery_unfound: setup_osds 5 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T16:50:16.372 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-04-12T16:50:16.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-04-12T16:50:16.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:50:16.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T16:50:16.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:50:16.380 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:50:16.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:50:16.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:50:16.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T16:50:16.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:50:16.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:50:16.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:50:16.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:50:16.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:50:16.382 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:50:16.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14e3a51a-1b60-4db8-b047-51589bb8bb96 2024-04-12T16:50:16.384 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 14e3a51a-1b60-4db8-b047-51589bb8bb96 2024-04-12T16:50:16.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 14e3a51a-1b60-4db8-b047-51589bb8bb96' 2024-04-12T16:50:16.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:50:16.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIZhlm/l9gGBAAw3qq+t5F86I0rOoKzGxsQw== 2024-04-12T16:50:16.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIZhlm/l9gGBAAw3qq+t5F86I0rOoKzGxsQw=="}' 2024-04-12T16:50:16.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14e3a51a-1b60-4db8-b047-51589bb8bb96 -i td/test-erasure-eio/0/new.json 2024-04-12T16:50:16.775 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:50:16.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T16:50:16.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIZhlm/l9gGBAAw3qq+t5F86I0rOoKzGxsQw== --osd-uuid 14e3a51a-1b60-4db8-b047-51589bb8bb96 2024-04-12T16:50:16.835 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:16.836+0000 7fc3f57697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:50:16.860 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:16.860+0000 7fc3f57697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:50:16.862 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:16.864+0000 7fc3f57697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:50:16.862 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:16.864+0000 7fc3f57697c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:50:16.863 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:16.864+0000 7fc3f57697c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:50:16.863 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:16.864+0000 7fc3f57697c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:50:16.900 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:16.900+0000 7fc3f57697c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T16:50:25.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T16:50:25.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:50:25.241 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:50:25.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:50:25.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:50:25.833 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:50:25.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:50:25.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:50:25.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:50:25.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:50:25.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:50:25.873 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:25.876+0000 7f89d74817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:50:25.924 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:25.924+0000 7f89d74817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:50:25.925 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:25.928+0000 7f89d74817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:50:26.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:50:26.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:50:26.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:50:26.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:50:26.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:50:26.254 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:26.255 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:50:26.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:50:26.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:26.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:26.508 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:26.508+0000 7f89d74817c0 -1 Falling back to public interface 2024-04-12T16:50:26.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:27.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:27.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:27.721 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:50:27.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:50:27.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:27.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:28.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:29.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:29.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:29.142 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:50:29.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:50:29.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:29.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:29.311 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:50:29.312+0000 7f89d74817c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:50:29.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:30.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:30.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:30.586 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:50:30.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:50:30.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:30.587 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:31.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:32.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:32.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:32.094 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:50:32.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:50:32.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:32.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:32.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:33.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:33.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:33.715 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:50:33.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:50:33.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:33.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:34.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:35.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:35.365 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:50:35.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:50:35.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:35.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:35.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:36.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:36.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:36.832 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:50:36.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:50:36.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:36.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:37.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:38.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:38.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:38.367 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:50:38.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:50:38.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:38.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:38.868 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:39.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:39.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:39.871 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:50:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:50:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:39.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:40.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:41.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:41.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:41.466 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:50:41.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:50:41.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:41.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:42.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:43.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:43.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:43.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:50:43.369 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:50:43.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:43.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:43.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:44.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:44.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:44.896 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:50:44.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:50:44.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:44.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:45.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:46.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:46.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:46.558 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:50:46.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:50:46.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:46.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:47.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:48.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:48.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:48.341 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:50:48.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:50:48.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:48.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:49.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:50.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:50.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:50.506 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:50:50.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:50:50.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:50.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:51.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:52.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:52.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:52.225 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:50:52.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:50:52.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:52.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:53.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:54.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:54.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:54.262 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:50:54.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:50:54.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:54.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:55.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:56.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:56.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:56.751 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:50:56.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:50:56.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:56.752 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:57.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:50:58.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:50:58.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:50:58.804 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:50:58.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:50:58.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:50:58.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:50:59.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:00.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:00.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:00.477 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:51:00.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:51:00.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:00.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:51:01.232 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3424862704,v1:127.0.0.1:6803/3424862704] [v2:127.0.0.1:6804/3424862704,v1:127.0.0.1:6805/3424862704] exists,up 14e3a51a-1b60-4db8-b047-51589bb8bb96 2024-04-12T16:51:01.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:51:01.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:51:01.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:51:01.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:51:01.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T16:51:01.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:51:01.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:51:01.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:51:01.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:51:01.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T16:51:01.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:51:01.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:51:01.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:51:01.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:51:01.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T16:51:01.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T16:51:01.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:51:01.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:51:01.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:51:01.244 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:51:01.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:51:01.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:51:01.245 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:51:01.245 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:51:01.245 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:51:01.245 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:51:01.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:51:01.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:51:01.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:51:01.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:51:01.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:51:01.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:51:01.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:51:01.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:51:01.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:51:01.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:51:01.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:51:01.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:51:01.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T16:51:01.248 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:51:01.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97afc880-7e8a-456c-9876-3b55e16f6df0 2024-04-12T16:51:01.249 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 97afc880-7e8a-456c-9876-3b55e16f6df0 2024-04-12T16:51:01.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 97afc880-7e8a-456c-9876-3b55e16f6df0' 2024-04-12T16:51:01.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:51:01.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1ZhlmP8RhEBAAzHA6P3yHr+QYTX4Q5LPphw== 2024-04-12T16:51:01.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1ZhlmP8RhEBAAzHA6P3yHr+QYTX4Q5LPphw=="}' 2024-04-12T16:51:01.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97afc880-7e8a-456c-9876-3b55e16f6df0 -i td/test-erasure-eio/1/new.json 2024-04-12T16:51:01.981 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:51:01.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T16:51:02.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1ZhlmP8RhEBAAzHA6P3yHr+QYTX4Q5LPphw== --osd-uuid 97afc880-7e8a-456c-9876-3b55e16f6df0 2024-04-12T16:51:02.043 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:02.044+0000 7fbb9aa6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:02.069 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:02.068+0000 7fbb9aa6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:02.070 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:02.072+0000 7fbb9aa6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:02.071 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:02.072+0000 7fbb9aa6b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:51:02.071 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:02.072+0000 7fbb9aa6b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:51:02.071 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:02.072+0000 7fbb9aa6b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:51:02.186 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:02.188+0000 7fbb9aa6b7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T16:51:09.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T16:51:09.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:51:09.765 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:51:09.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:51:09.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:51:10.371 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:51:10.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:51:10.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:51:10.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:51:10.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:51:10.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:51:10.411 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:10.412+0000 7febade1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:10.458 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:10.460+0000 7febade1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:10.460 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:10.460+0000 7febade1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:10.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:51:10.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:51:10.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:51:10.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:51:10.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:51:10.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:10.793 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:51:10.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:51:10.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:10.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:11.044 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:11.044+0000 7febade1d7c0 -1 Falling back to public interface 2024-04-12T16:51:11.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:12.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:12.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:12.264 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:51:12.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:51:12.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:12.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:12.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:13.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:13.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:13.688 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:51:13.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:51:13.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:13.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:13.891 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:13.892+0000 7febade1d7c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:51:14.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:15.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:15.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:15.152 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:51:15.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:51:15.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:15.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:15.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:16.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:16.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:16.665 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:51:16.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:51:16.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:16.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:17.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:18.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:18.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:18.299 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:51:18.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:51:18.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:18.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:18.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:19.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:19.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:19.941 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:51:19.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:51:19.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:19.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:20.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:21.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:21.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:21.589 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:51:21.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:51:21.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:21.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:22.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:23.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:23.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:23.154 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:51:23.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:51:23.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:23.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:23.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:24.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:24.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:24.615 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:51:24.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:51:24.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:24.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:25.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:26.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:26.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:26.238 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:51:26.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:51:26.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:26.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:26.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:27.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:27.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:27.731 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:51:27.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:51:27.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:27.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:28.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:29.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:29.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:29.327 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:51:29.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:51:29.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:29.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:29.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:30.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:30.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:30.879 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:51:30.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:51:30.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:30.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:31.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:32.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:32.455 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:32.456 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:51:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:51:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:32.456 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:32.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:33.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:33.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:33.916 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:51:33.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:51:33.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:33.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:35.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:36.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:36.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:36.219 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:51:36.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:51:36.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:36.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:37.357 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:38.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:38.359 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:51:38.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:38.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:51:38.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:38.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:39.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:40.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:40.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:40.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:51:40.846 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:51:40.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:40.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:41.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:42.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:42.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:42.688 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:51:42.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:51:42.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:42.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:43.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:44.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:44.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:44.828 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:51:44.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:51:44.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:44.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:45.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:46.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:46.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:46.862 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:51:46.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:51:46.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:46.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:51:47.723 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1593433208,v1:127.0.0.1:6811/1593433208] [v2:127.0.0.1:6812/1593433208,v1:127.0.0.1:6813/1593433208] exists,up 97afc880-7e8a-456c-9876-3b55e16f6df0 2024-04-12T16:51:47.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:51:47.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:51:47.727 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:51:47.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:51:47.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:51:47.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:51:47.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:51:47.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:51:47.730 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:51:47.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=921f1db7-2de1-425c-8892-e5dbb731553b 2024-04-12T16:51:47.731 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 921f1db7-2de1-425c-8892-e5dbb731553b 2024-04-12T16:51:47.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 921f1db7-2de1-425c-8892-e5dbb731553b' 2024-04-12T16:51:47.731 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:51:47.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjZhlmwkeSLRAAaklu0g6Jg0bw51lp/vzkrQ== 2024-04-12T16:51:47.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjZhlmwkeSLRAAaklu0g6Jg0bw51lp/vzkrQ=="}' 2024-04-12T16:51:47.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 921f1db7-2de1-425c-8892-e5dbb731553b -i td/test-erasure-eio/2/new.json 2024-04-12T16:51:48.446 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:51:48.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T16:51:48.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjZhlmwkeSLRAAaklu0g6Jg0bw51lp/vzkrQ== --osd-uuid 921f1db7-2de1-425c-8892-e5dbb731553b 2024-04-12T16:51:48.510 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:48.508+0000 7f44119817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:48.548 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:48.548+0000 7f44119817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:48.550 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:48.548+0000 7f44119817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:48.550 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:48.548+0000 7f44119817c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:51:48.550 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:48.548+0000 7f44119817c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:51:48.550 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:48.548+0000 7f44119817c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:51:48.588 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:48.588+0000 7f44119817c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T16:51:56.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T16:51:56.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:51:56.638 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:51:56.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:51:56.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:51:57.232 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:51:57.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:51:57.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:51:57.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:51:57.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:51:57.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:51:57.272 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:57.272+0000 7fa4232187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:57.322 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:57.320+0000 7fa4232187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:57.324 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:57.324+0000 7fa4232187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:51:57.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:51:57.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:51:57.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:51:57.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:51:57.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:51:57.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:57.692 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:51:57.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:51:57.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:57.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:51:57.900 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:51:57.900+0000 7fa4232187c0 -1 Falling back to public interface 2024-04-12T16:51:58.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:51:59.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:51:59.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:51:59.158 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:51:59.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:51:59.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:51:59.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:51:59.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:00.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:00.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:00.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:52:00.585 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:52:00.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:00.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:00.821 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:00.820+0000 7fa4232187c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:52:01.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:02.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:02.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:02.134 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:52:02.134 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:52:02.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:02.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:02.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:03.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:03.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:03.789 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:52:03.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:52:03.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:03.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:04.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:05.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:05.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:05.367 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:52:05.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:52:05.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:05.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:05.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:06.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:06.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:06.849 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:52:06.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:52:06.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:06.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:07.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:08.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:08.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:08.368 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:52:08.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:52:08.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:08.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:08.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:09.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:09.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:09.940 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:52:09.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:52:09.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:09.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:10.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:11.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:11.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:11.583 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:52:11.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:52:11.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:11.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:12.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:13.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:13.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:13.161 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:52:13.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:52:13.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:13.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:13.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:14.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:14.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:14.704 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:52:14.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:52:14.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:14.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:15.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:16.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:16.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:16.286 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:52:16.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:52:16.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:16.287 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:16.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:17.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:17.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:17.817 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:52:17.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:52:17.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:17.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:18.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:19.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:19.354 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:52:19.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:19.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:52:19.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:19.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:20.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:21.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:21.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:21.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:52:21.012 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:52:21.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:21.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:21.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:22.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:22.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:22.736 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:52:22.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:52:22.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:22.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:23.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:24.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:24.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:24.750 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:52:24.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:52:24.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:24.751 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:25.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:26.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:26.820 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:26.821 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:52:26.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:52:26.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:26.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:27.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:28.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:28.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:28.722 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:52:28.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:52:28.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:28.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:29.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:30.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:30.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:30.550 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:52:30.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:52:30.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:30.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:31.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:32.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:32.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:32.407 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:52:32.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:52:32.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:32.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:32.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:33.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:33.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:33.955 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:52:33.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:52:33.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:33.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:52:34.651 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2861021590,v1:127.0.0.1:6819/2861021590] [v2:127.0.0.1:6820/2861021590,v1:127.0.0.1:6821/2861021590] exists,up 921f1db7-2de1-425c-8892-e5dbb731553b 2024-04-12T16:52:34.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:52:34.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:52:34.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:52:34.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T16:52:34.658 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:52:34.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb147f4f-8a32-47e7-9eb5-e8b3a1cb12fe 2024-04-12T16:52:34.658 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 cb147f4f-8a32-47e7-9eb5-e8b3a1cb12fe 2024-04-12T16:52:34.658 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 cb147f4f-8a32-47e7-9eb5-e8b3a1cb12fe' 2024-04-12T16:52:34.658 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:52:34.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSZhlmEbBhKRAAdosD7RswuJwLarFuLgVKiw== 2024-04-12T16:52:34.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSZhlmEbBhKRAAdosD7RswuJwLarFuLgVKiw=="}' 2024-04-12T16:52:34.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb147f4f-8a32-47e7-9eb5-e8b3a1cb12fe -i td/test-erasure-eio/3/new.json 2024-04-12T16:52:35.870 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:52:35.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T16:52:35.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSZhlmEbBhKRAAdosD7RswuJwLarFuLgVKiw== --osd-uuid cb147f4f-8a32-47e7-9eb5-e8b3a1cb12fe 2024-04-12T16:52:35.935 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:35.931+0000 7f24c55287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:52:35.937 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:35.935+0000 7f24c55287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:52:35.938 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:35.935+0000 7f24c55287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:52:35.939 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:35.935+0000 7f24c55287c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:52:35.939 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:35.935+0000 7f24c55287c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:52:35.939 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:35.935+0000 7f24c55287c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:52:36.186 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:36.187+0000 7f24c55287c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T16:52:44.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T16:52:44.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:52:44.051 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T16:52:44.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T16:52:44.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:52:44.639 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:52:44.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T16:52:44.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:52:44.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:52:44.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:52:44.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:52:44.679 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:44.675+0000 7f5879ddc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:52:44.720 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:44.719+0000 7f5879ddc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:52:44.722 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:44.719+0000 7f5879ddc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:52:45.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T16:52:45.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:52:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:52:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:52:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:52:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:45.063 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:52:45.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:52:45.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:45.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:45.286 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:45.283+0000 7f5879ddc7c0 -1 Falling back to public interface 2024-04-12T16:52:45.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:46.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:46.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:46.607 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:52:46.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:52:46.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:46.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:47.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:48.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:48.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:48.044 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:52:48.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:52:48.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:48.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:48.275 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:52:48.271+0000 7f5879ddc7c0 -1 osd.3 0 log_to_monitors true 2024-04-12T16:52:48.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:49.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:49.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:49.577 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:52:49.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:52:49.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:49.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:50.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:51.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:51.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:51.324 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:52:51.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:52:51.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:51.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:52.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:53.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:53.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:53.084 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:52:53.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:52:53.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:53.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:53.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:54.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:54.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:54.574 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:52:54.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:52:54.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:54.575 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:55.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:56.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:56.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:56.216 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:52:56.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:52:56.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:56.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:56.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:57.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:57.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:57.809 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:52:57.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:52:57.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:57.810 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:58.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:52:59.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:52:59.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:52:59.405 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:52:59.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:52:59.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:52:59.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:52:59.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:00.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:00.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:00.887 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:53:00.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:53:00.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:00.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:01.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:02.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:02.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:02.438 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:53:02.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:53:02.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:02.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:03.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:04.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:04.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:04.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:53:04.160 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:53:04.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:04.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:04.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:05.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:05.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:05.883 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:53:05.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:53:05.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:05.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:06.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:07.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:07.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:07.560 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:53:07.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:53:07.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:07.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:08.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:09.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:09.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:09.047 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:53:09.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:53:09.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:09.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:10.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:11.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:11.295 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:11.296 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:53:11.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:53:11.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:11.296 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:12.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:13.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:13.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:13.268 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:53:13.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:53:13.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:13.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:14.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:15.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:15.655 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:15.655 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:53:15.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:53:15.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:15.656 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:16.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:17.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:17.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:17.976 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:53:17.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:53:17.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:17.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:18.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:19.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:19.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:19.960 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:53:19.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:53:19.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:19.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:21.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:22.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:22.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:22.165 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:53:22.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:53:22.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:22.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:53:22.659 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3857400375,v1:127.0.0.1:6827/3857400375] [v2:127.0.0.1:6828/3857400375,v1:127.0.0.1:6829/3857400375] exists,up cb147f4f-8a32-47e7-9eb5-e8b3a1cb12fe 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:53:22.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:53:22.664 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:53:22.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-04-12T16:53:22.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:53:22.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e3b30110-f336-4f42-b1c1-ae32de40fd75 2024-04-12T16:53:22.666 INFO:tasks.workunit.client.0.smithi137.stdout:add osd4 e3b30110-f336-4f42-b1c1-ae32de40fd75 2024-04-12T16:53:22.666 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 e3b30110-f336-4f42-b1c1-ae32de40fd75' 2024-04-12T16:53:22.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:53:22.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACZxlmuxiqKRAAPts7k9wMtLM0a0rsrwvaIA== 2024-04-12T16:53:22.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACZxlmuxiqKRAAPts7k9wMtLM0a0rsrwvaIA=="}' 2024-04-12T16:53:22.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e3b30110-f336-4f42-b1c1-ae32de40fd75 -i td/test-erasure-eio/4/new.json 2024-04-12T16:53:24.209 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:53:24.228 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-04-12T16:53:24.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACZxlmuxiqKRAAPts7k9wMtLM0a0rsrwvaIA== --osd-uuid e3b30110-f336-4f42-b1c1-ae32de40fd75 2024-04-12T16:53:24.274 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:24.275+0000 7fc3fc70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:53:24.323 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:24.323+0000 7fc3fc70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:53:24.325 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:24.327+0000 7fc3fc70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:53:24.325 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:24.327+0000 7fc3fc70b7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:53:24.325 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:24.327+0000 7fc3fc70b7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:53:24.325 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:24.327+0000 7fc3fc70b7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:53:24.657 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:24.659+0000 7fc3fc70b7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-04-12T16:53:32.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-04-12T16:53:32.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:53:32.964 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd4 key to auth repository 2024-04-12T16:53:32.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-12T16:53:32.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:53:33.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-12T16:53:33.562 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.4 2024-04-12T16:53:33.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:53:33.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:53:33.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:53:33.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:53:33.604 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:33.607+0000 7f6817af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:53:33.642 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:33.643+0000 7f6817af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:53:33.643 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:33.643+0000 7f6817af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:53:33.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-12T16:53:33.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:53:33.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-12T16:53:33.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:53:33.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:53:33.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:33.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:53:33.990 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:53:33.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:33.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:34.225 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:34.227+0000 7f6817af27c0 -1 Falling back to public interface 2024-04-12T16:53:34.457 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:35.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:35.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:35.459 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:53:35.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:53:35.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:35.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:35.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:36.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:36.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:36.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:53:36.914 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:53:36.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:36.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:37.095 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:53:37.095+0000 7f6817af27c0 -1 osd.4 0 log_to_monitors true 2024-04-12T16:53:37.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:38.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:38.395 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:53:38.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:53:38.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:38.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:39.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:40.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:40.503 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:40.504 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:53:40.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:53:40.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:40.504 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:41.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:42.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:42.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:42.361 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:53:42.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:53:42.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:42.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:42.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:43.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:43.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:43.898 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:53:43.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:53:43.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:43.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:44.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:45.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:45.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:45.391 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:53:45.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:53:45.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:45.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:45.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:46.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:46.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:46.854 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:53:46.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:53:46.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:46.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:47.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:48.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:48.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:48.444 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:53:48.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:53:48.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:48.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:49.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:50.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:50.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:50.057 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:53:50.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:53:50.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:50.058 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:50.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:51.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:51.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:51.713 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:53:51.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:53:51.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:51.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:52.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:53.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:53.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:53.301 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:53:53.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:53:53.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:53.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:53.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:54.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:54.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:54.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:53:54.881 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:53:54.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:54.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:55.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:56.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:56.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:56.470 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:53:56.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:53:56.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:56.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:57.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:58.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:58.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:58.177 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:53:58.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:53:58.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:58.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:53:58.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:53:59.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:53:59.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:53:59.911 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:53:59.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:53:59.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:53:59.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:54:00.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:54:01.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:54:01.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:54:01.482 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:54:01.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:54:01.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:54:01.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:54:02.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:54:03.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:54:03.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:54:03.839 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:54:03.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:54:03.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:54:03.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:54:04.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:54:05.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:54:05.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:54:05.447 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:54:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:54:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:54:05.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:54:06.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:54:07.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:54:07.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:54:07.151 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:54:07.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:54:07.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:54:07.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:54:08.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:54:09.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:54:09.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:54:09.102 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:54:09.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:54:09.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:54:09.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:54:09.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:54:10.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:54:10.933 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:54:10.933 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:54:10.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:54:10.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:54:10.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:54:11.451 INFO:tasks.workunit.client.0.smithi137.stdout:osd.4 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2221133342,v1:127.0.0.1:6835/2221133342] [v2:127.0.0.1:6836/2221133342,v1:127.0.0.1:6837/2221133342] exists,up e3b30110-f336-4f42-b1c1-ae32de40fd75 2024-04-12T16:54:11.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:54:11.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:54:11.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:54:11.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T16:54:11.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T16:54:11.452 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T16:54:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:54:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:54:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:54:11.453 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T16:54:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T16:54:11.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T16:54:11.561 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:54:11.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T16:54:11.579 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:50:26.516+0000 7f89d74817c0 0 load: jerasure load: lrc 2024-04-12T16:54:11.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:621: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T16:54:11.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-04-12T16:54:12.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T16:54:12.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T16:54:14.389 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T16:54:14.407 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:54:15.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T16:54:15.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:54:15.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:54:15.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:54:15.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:54:15.411 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:54:15.411 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:54:15.411 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:54:15.411 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:54:15.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:54:15.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:54:15.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:54:15.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:54:15.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:54:15.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:54:15.540 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:54:16.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:54:16.063 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:54:16.063 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:54:16.063 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:54:16.063 INFO:tasks.workunit.client.0.smithi137.stderr:4' 2024-04-12T16:54:16.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:54:16.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:54:16.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:54:16.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803825 2024-04-12T16:54:16.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803825 2024-04-12T16:54:16.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825' 2024-04-12T16:54:16.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:54:16.227 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:54:16.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607592 2024-04-12T16:54:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607592 2024-04-12T16:54:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-51539607592' 2024-04-12T16:54:16.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:54:16.479 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:54:16.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411358 2024-04-12T16:54:16.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411358 2024-04-12T16:54:16.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-51539607592 2-77309411358' 2024-04-12T16:54:16.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:54:16.701 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:54:16.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117012 2024-04-12T16:54:16.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117012 2024-04-12T16:54:16.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-51539607592 2-77309411358 3-115964117012' 2024-04-12T16:54:16.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:54:16.857 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:54:17.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789962 2024-04-12T16:54:17.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789962 2024-04-12T16:54:17.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-51539607592 2-77309411358 3-115964117012 4-158913789962' 2024-04-12T16:54:17.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:54:17.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803825 2024-04-12T16:54:17.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:54:17.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:54:17.011 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803825 2024-04-12T16:54:17.011 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:54:17.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803825 2024-04-12T16:54:17.012 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803825 2024-04-12T16:54:17.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-04-12T16:54:17.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:54:17.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803822 -lt 25769803825 2024-04-12T16:54:17.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T16:54:18.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T16:54:18.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:54:18.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803825 -lt 25769803825 2024-04-12T16:54:18.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:54:18.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:54:18.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607592 2024-04-12T16:54:18.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:54:18.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607592 2024-04-12T16:54:18.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:54:18.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607592 2024-04-12T16:54:18.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607592' 2024-04-12T16:54:18.865 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 51539607592 2024-04-12T16:54:18.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:54:19.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607593 -lt 51539607592 2024-04-12T16:54:19.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:54:19.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:54:19.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411358 2024-04-12T16:54:19.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:54:19.270 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411358 2024-04-12T16:54:19.270 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:54:19.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411358 2024-04-12T16:54:19.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411358' 2024-04-12T16:54:19.271 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 77309411358 2024-04-12T16:54:19.272 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:54:19.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411359 -lt 77309411358 2024-04-12T16:54:19.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:54:19.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117012 2024-04-12T16:54:19.756 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:54:19.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:54:19.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117012 2024-04-12T16:54:19.758 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:54:19.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117012 2024-04-12T16:54:19.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117012' 2024-04-12T16:54:19.760 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 115964117012 2024-04-12T16:54:19.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:54:20.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117013 -lt 115964117012 2024-04-12T16:54:20.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:54:20.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-158913789962 2024-04-12T16:54:20.249 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:54:20.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:54:20.252 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-158913789962 2024-04-12T16:54:20.252 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:54:20.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789962 2024-04-12T16:54:20.253 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 158913789962' 2024-04-12T16:54:20.254 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 158913789962 2024-04-12T16:54:20.254 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:54:20.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789962 -lt 158913789962 2024-04-12T16:54:20.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:54:20.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:54:20.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:54:21.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:54:21.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:54:21.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:54:21.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:54:21.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:54:21.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:54:21.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:54:21.230 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:54:21.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:54:21.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:54:21.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:54:21.629 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:54:22.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:54:22.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:54:22.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:54:22.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-04-12T16:54:22.574 INFO:tasks.workunit.client.0.smithi137.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-12T16:54:22.574 INFO:tasks.workunit.client.0.smithi137.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:16.001079+0000 0'0 42:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-04-12T16:54:13.844588+0000 0'0 2024-04-12T16:54:13.844588+0000 0 0 periodic scrub scheduled @ 2024-04-13T17:32:46.325186+0000 0 0 2024-04-12T16:54:22.574 INFO:tasks.workunit.client.0.smithi137.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:13.565030+0000 0'0 42:17 [4,1,2] 4 [4,1,2] 4 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T16:57:43.336300+0000 0 0 2024-04-12T16:54:22.574 INFO:tasks.workunit.client.0.smithi137.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:13.503720+0000 0'0 43:100 [0,4,1] 0 [0,4,1] 0 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T16:52:54.070116+0000 0 0 2024-04-12T16:54:22.574 INFO:tasks.workunit.client.0.smithi137.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:13.453825+0000 0'0 42:17 [4,3,0] 4 [4,3,0] 4 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T20:09:46.868570+0000 0 0 2024-04-12T16:54:22.574 INFO:tasks.workunit.client.0.smithi137.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:52:35.480041+0000 0'0 43:70 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T21:02:51.354357+0000 0 0 2024-04-12T16:54:22.574 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-04-12T16:54:22.575 INFO:tasks.workunit.client.0.smithi137.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-12T16:54:22.575 INFO:tasks.workunit.client.0.smithi137.stderr:dumped pgs 2024-04-12T16:54:22.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T16:54:22.590 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:22.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-04-12T16:54:22.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:54:22.826 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T16:54:22.827 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T16:54:22.827 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:54:23.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:54:23.259 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:54:23.259 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:54:23.260 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:54:23.260 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T16:54:23.260 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-04-12T16:54:23.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-04-12T16:54:23.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: local -a initial_osds 2024-04-12T16:54:23.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local last_osd=2 2024-04-12T16:54:23.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-04-12T16:54:23.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:54:23.261 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:54:23.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:54:23.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:54:23.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:54:23.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:54:23.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: ceph osd down 2 2024-04-12T16:54:24.524 INFO:tasks.workunit.client.0.smithi137.stderr:osd.2 is already down. 2024-04-12T16:54:24.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: ceph osd out 2 2024-04-12T16:54:25.677 INFO:tasks.workunit.client.0.smithi137.stderr:osd.2 is already out. 2024-04-12T16:54:25.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:16.001079+0000 0'0 43:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-04-12T16:54:13.844588+0000 0'0 2024-04-12T16:54:13.844588+0000 0 0 periodic scrub scheduled @ 2024-04-13T21:28:47.701227+0000 0 0 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:13.565030+0000 0'0 43:19 [4,1,2] 4 [4,1,2] 4 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T16:57:43.336300+0000 0 0 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:13.503720+0000 0'0 43:100 [0,4,1] 0 [0,4,1] 0 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T16:52:54.070116+0000 0 0 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:13.453825+0000 0'0 43:19 [4,3,0] 4 [4,3,0] 4 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T20:09:46.868570+0000 0 0 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:52:35.480041+0000 0'0 43:70 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T21:02:51.354357+0000 0 0 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-12T16:54:26.091 INFO:tasks.workunit.client.0.smithi137.stderr:dumped pgs 2024-04-12T16:54:26.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-04-12T16:54:26.107 INFO:tasks.workunit.client.0.smithi137.stderr:4+0 records in 2024-04-12T16:54:26.107 INFO:tasks.workunit.client.0.smithi137.stderr:4+0 records out 2024-04-12T16:54:26.108 INFO:tasks.workunit.client.0.smithi137.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000169828 s, 24.1 MB/s 2024-04-12T16:54:26.108 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: seq 1 100 2024-04-12T16:54:26.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:26.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:26.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:26.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:26.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:26.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:27.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:27.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:28.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:28.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:28.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:28.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:28.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:28.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:29.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:29.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:29.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:29.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:29.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:29.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:30.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:30.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:30.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:30.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:30.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:30.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:31.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:31.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:31.462 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:31.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:31.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:31.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:32.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:32.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:32.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:32.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:32.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:32.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:33.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:33.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:33.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:33.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:33.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:33.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:34.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:34.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:34.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:34.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:34.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:34.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:35.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:35.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:35.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:35.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:35.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:35.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:36.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:36.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:36.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:36.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:36.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:36.818 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:37.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:37.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:37.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:37.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:37.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:37.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:38.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:38.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:38.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:38.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:38.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:38.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:39.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:39.255 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:39.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:39.649 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:40.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:40.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:40.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:40.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:40.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:40.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:41.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:41.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:41.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:41.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:41.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:41.702 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:42.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:42.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:42.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:42.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:42.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:42.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:43.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:43.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:43.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:43.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:43.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:43.839 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:44.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:44.198 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:44.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:44.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:44.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:44.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:45.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:45.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:45.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:45.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:45.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:45.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:46.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:46.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:46.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:46.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:46.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:46.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:47.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:47.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:47.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:47.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:47.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:47.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:48.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:48.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:48.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:48.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:49.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:49.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:49.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:49.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:49.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:49.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:50.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:50.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:50.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:50.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:50.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:50.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:51.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:51.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:51.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:51.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:52.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:52.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:52.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:52.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:52.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:52.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:53.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:53.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:53.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:53.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:53.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:53.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:54.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:54.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:54.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:54.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:54.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:54.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:55.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:55.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:55.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:55.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:55.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:55.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:56.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:56.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:56.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:56.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:56.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:56.701 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:57.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:57.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:57.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:57.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:57.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:57.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:58.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:58.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:58.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:58.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:58.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:58.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:59.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:59.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:59.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:59.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-04-12T16:54:59.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:54:59.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-04-12T16:55:00.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:00.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-04-12T16:55:00.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:00.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-04-12T16:55:00.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:00.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-04-12T16:55:01.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-04-12T16:55:01.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:55:01.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj75 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj75 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:55:01.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-12T16:55:01.078 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-04-12T16:55:01.078 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:55:01.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:55:01.500 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:55:01.500 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:55:01.500 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:55:01.500 INFO:tasks.workunit.client.0.smithi137.stderr:2147483647' 2024-04-12T16:55:01.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-04-12T16:55:01.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-04-12T16:55:01.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:55:01.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-04-12T16:55:01.501 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:55:01.501 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-04-12T16:55:01.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:55:01.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-12T16:55:01.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:55:01.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-12T16:55:01.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:55:01.502 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:55:01.503 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:55:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-12T16:55:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T16:55:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T16:55:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:55:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:01.504 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T16:55:01.504 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:55:01.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:55:01.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:55:01.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:55:01.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-04-12T16:55:01.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T16:55:01.619 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T16:55:01.620 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:55:01.620 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:01.620 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:01.620 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T16:55:01.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:55:01.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-04-12T16:55:01.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:643: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-04-12T16:55:01.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T16:55:01.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj75 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj75 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T16:55:01.730 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-12T16:55:01.731 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-04-12T16:55:01.731 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr:2147483647' 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T16:55:02.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-04-12T16:55:02.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T16:55:02.153 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-04-12T16:55:02.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T16:55:02.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-12T16:55:02.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T16:55:02.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-12T16:55:02.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T16:55:02.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T16:55:02.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T16:55:02.154 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-12T16:55:02.154 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-12T16:55:02.154 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-12T16:55:02.155 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:55:02.155 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:02.155 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:02.155 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.1.asok 2024-04-12T16:55:02.155 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-12T16:55:02.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T16:55:02.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T16:55:02.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T16:55:02.278 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-04-12T16:55:02.278 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-12T16:55:02.278 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-12T16:55:02.278 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:55:02.278 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:02.279 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:02.279 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.1.asok 2024-04-12T16:55:02.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T16:55:02.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-04-12T16:55:02.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-04-12T16:55:02.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T16:55:02.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T16:55:02.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-12T16:55:02.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:55:02.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:55:02.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:55:02.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T16:55:02.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T16:55:02.401 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:55:02.401 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:55:02.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-12T16:55:02.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:55:02.403 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-04-12T16:55:02.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-12T16:55:02.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T16:55:02.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T16:55:02.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T16:55:02.438 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:02.439+0000 7f79a45557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:02.440 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:02.439+0000 7f79a45557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:02.442 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:02.443+0000 7f79a45557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:02.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-12T16:55:02.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:55:02.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:55:02.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:55:02.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:55:02.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:55:02.799 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:55:02.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:55:02.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:55:02.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:55:02.990 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:02.991+0000 7f79a45557c0 -1 Falling back to public interface 2024-04-12T16:55:03.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:55:04.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:55:04.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:55:04.277 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:55:04.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:55:04.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:55:04.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:55:04.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:55:05.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:55:05.689 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:55:05.689 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:55:05.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:55:05.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:55:05.690 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:55:06.256 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:55:06.966 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:06.967+0000 7f79a45557c0 -1 osd.2 43 log_to_monitors true 2024-04-12T16:55:07.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:55:07.258 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:55:07.258 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:55:07.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:55:07.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:55:07.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:55:07.920 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up out weight 0 up_from 50 up_thru 20 down_at 44 last_clean_interval [18,43) [v2:127.0.0.1:6818/2615906416,v1:127.0.0.1:6819/2615906416] [v2:127.0.0.1:6820/2615906416,v1:127.0.0.1:6821/2615906416] exists,up 921f1db7-2de1-425c-8892-e5dbb731553b 2024-04-12T16:55:07.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:55:07.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:55:07.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:55:07.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: ceph osd in 2 2024-04-12T16:55:09.458 INFO:tasks.workunit.client.0.smithi137.stderr:osd.2 is already in. 2024-04-12T16:55:09.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: sleep 15 2024-04-12T16:55:24.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: seq 1 100 2024-04-12T16:55:24.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-04-12T16:55:24.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: get_state 2.0 2024-04-12T16:55:24.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-04-12T16:55:24.484 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-04-12T16:55:24.485 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-04-12T16:55:24.485 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-12T16:55:24.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-04-12T16:55:24.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-04-12T16:55:24.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: grep recovery_unfound 2024-04-12T16:55:24.905 INFO:tasks.workunit.client.0.smithi137.stdout:active+recovery_unfound+degraded 2024-04-12T16:55:24.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-04-12T16:55:24.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: break 2024-04-12T16:55:24.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:660: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-04-12T16:55:25.316 INFO:tasks.workunit.client.0.smithi137.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-12T16:55:25.316 INFO:tasks.workunit.client.0.smithi137.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-04-12T16:55:14.842264+0000 48'101 53:345 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-04-12T16:54:13.844588+0000 0'0 2024-04-12T16:54:13.844588+0000 0 0 no scrub is scheduled 0 0 2024-04-12T16:55:25.317 INFO:tasks.workunit.client.0.smithi137.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:55:10.529078+0000 0'0 53:51 [4,1,2] 4 [4,1,2] 4 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-14T02:04:49.579004+0000 0 0 2024-04-12T16:55:25.317 INFO:tasks.workunit.client.0.smithi137.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:13.503720+0000 0'0 53:120 [0,4,1] 0 [0,4,1] 0 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T16:52:54.070116+0000 0 0 2024-04-12T16:55:25.317 INFO:tasks.workunit.client.0.smithi137.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:54:13.453825+0000 0'0 53:38 [4,3,0] 4 [4,3,0] 4 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T20:09:46.868570+0000 0 0 2024-04-12T16:55:25.317 INFO:tasks.workunit.client.0.smithi137.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-12T16:55:10.529507+0000 0'0 53:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-12T16:50:15.066613+0000 0'0 2024-04-12T16:50:15.066613+0000 0 0 periodic scrub scheduled @ 2024-04-13T17:36:08.747976+0000 0 0 2024-04-12T16:55:25.317 INFO:tasks.workunit.client.0.smithi137.stdout: 2024-04-12T16:55:25.317 INFO:tasks.workunit.client.0.smithi137.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-12T16:55:25.317 INFO:tasks.workunit.client.0.smithi137.stderr:dumped pgs 2024-04-12T16:55:25.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:661: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:55:25.479 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-04-12T16:55:25.480 INFO:tasks.workunit.client.0.smithi137.stdout: "num_missing": 1, 2024-04-12T16:55:25.480 INFO:tasks.workunit.client.0.smithi137.stdout: "num_unfound": 1, 2024-04-12T16:55:25.480 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": [ 2024-04-12T16:55:25.480 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.480 INFO:tasks.workunit.client.0.smithi137.stdout: "oid": { 2024-04-12T16:55:25.480 INFO:tasks.workunit.client.0.smithi137.stdout: "oid": "obj75", 2024-04-12T16:55:25.480 INFO:tasks.workunit.client.0.smithi137.stdout: "key": "", 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "snapid": -2, 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "hash": 3565005477, 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "max": 0, 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "pool": 2, 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "namespace": "" 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "need": "48'76", 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "have": "0'0", 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "flags": "none", 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "locations": [ 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:55:25.481 INFO:tasks.workunit.client.0.smithi137.stdout: ] 2024-04-12T16:55:25.482 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.482 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.482 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "NotRecovering", 2024-04-12T16:55:25.482 INFO:tasks.workunit.client.0.smithi137.stdout: "available_might_have_unfound": true, 2024-04-12T16:55:25.482 INFO:tasks.workunit.client.0.smithi137.stdout: "might_have_unfound": [], 2024-04-12T16:55:25.482 INFO:tasks.workunit.client.0.smithi137.stdout: "more": false 2024-04-12T16:55:25.482 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-04-12T16:55:25.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-04-12T16:55:25.686 INFO:tasks.workunit.client.0.smithi137.stdout:{ 2024-04-12T16:55:25.686 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_trimq": "[]", 2024-04-12T16:55:25.686 INFO:tasks.workunit.client.0.smithi137.stdout: "snap_trimq_len": 0, 2024-04-12T16:55:25.686 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+recovery_unfound+degraded", 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch": 53, 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.687 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_recovery_backfill": [ 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "1(1)", 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "2(4)", 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "3(0)", 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "info": { 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s0", 2024-04-12T16:55:25.688 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "0", 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "48'101", 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "48'75", 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "0'0", 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 101, 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 41, 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 41, 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 51, 2024-04-12T16:55:25.689 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 41, 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 51, 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 51, 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 41, 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.690 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "48'101", 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 346, 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 53, 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+recovery_unfound+degraded", 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:55:14.914080+0000", 2024-04-12T16:55:25.691 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:55:14.842264+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:55:14.914080+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:55:14.914080+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:55:10.657209+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:55:10.657209+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:55:14.914080+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:55:10.082236+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:55:14.914080+0000", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 51, 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "0'0", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "0'0", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 41, 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:55:25.692 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 101, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 0, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 101, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:55:25.693 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "no scrub is scheduled", 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:55:25.694 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 413696, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 101, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 505, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 1, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 3, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 1, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 101, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:55:25.695 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 101, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 404, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 99, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 405504, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:55:25.696 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:55:25.697 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.698 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [ 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [ 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 100 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),4(2)", 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 1 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.699 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.700 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "peer_info": [ 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "peer": "0(3)", 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s3", 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "3", 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "48'101", 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "48'101", 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "0'0", 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 101, 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:55:25.701 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 41, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 41, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 51, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 41, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 51, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 51, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 41, 2024-04-12T16:55:25.702 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "48'101", 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 131, 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 48, 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized+degraded", 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.703 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:54:24.716957+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:54:24.716708+0000", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 51, 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "0'0", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "0'0", 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 41, 2024-04-12T16:55:25.704 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 101, 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 0, 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 101, 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:55:25.705 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-04-13T20:48:00.998122+0000", 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 413696, 2024-04-12T16:55:25.706 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 101, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 505, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 101, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 101, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:55:25.707 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 101, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 404, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:55:25.708 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:55:25.709 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:55:25.709 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:55:25.709 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:55:25.709 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:55:25.709 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:55:25.709 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:55:25.709 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:55:25.709 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:55:25.710 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:55:25.710 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:55:25.710 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:55:25.710 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.710 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:55:25.710 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.710 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.711 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.711 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.711 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.711 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.711 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:55:25.711 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.712 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.712 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.712 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.712 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.712 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.712 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [ 2024-04-12T16:55:25.712 INFO:tasks.workunit.client.0.smithi137.stdout: "3(0)", 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "1(1)", 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [ 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 101 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:55:25.713 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.714 INFO:tasks.workunit.client.0.smithi137.stdout: "peer": "1(1)", 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s1", 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "1", 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "48'101", 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "48'101", 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "0'0", 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 101, 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 41, 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 41, 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 51, 2024-04-12T16:55:25.715 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 41, 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 51, 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 51, 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 41, 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.716 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "48'101", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 131, 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 48, 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized+degraded", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:54:24.716957+0000", 2024-04-12T16:55:25.717 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:54:24.716708+0000", 2024-04-12T16:55:25.718 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 51, 2024-04-12T16:55:25.718 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "0'0", 2024-04-12T16:55:25.718 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "0'0", 2024-04-12T16:55:25.718 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 41, 2024-04-12T16:55:25.718 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.718 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:55:25.718 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:55:25.718 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 101, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 0, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 101, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:55:25.719 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-04-13T20:48:00.998122+0000", 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 413696, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 101, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 505, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 1, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 101, 2024-04-12T16:55:25.720 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 101, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 101, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 404, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-04-12T16:55:25.721 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:55:25.722 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.723 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [ 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: "3(0)", 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: "1(1)", 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [ 2024-04-12T16:55:25.724 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 101 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:55:25.725 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "peer": "2(4)", 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s4", 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "4", 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "48'101", 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "43'1", 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "0'0", 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 1, 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:55:25.726 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 41, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 41, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 51, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 41, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 51, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 51, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 41, 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.727 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "43'1", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 17, 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 43, 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+clean", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:54:16.001079+0000", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:54:15.999920+0000", 2024-04-12T16:55:25.728 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:54:15.999920+0000", 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 51, 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "0'0", 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "0'0", 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 41, 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:55:25.729 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 1, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 0, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 1, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:55:25.730 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-04-13T21:28:47.701227+0000", 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 4096, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 1, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 5, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-04-12T16:55:25.731 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 1, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 1, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 1, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 4, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-04-12T16:55:25.732 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:55:25.733 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:55:25.734 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.735 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.735 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.735 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.735 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.735 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.735 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:55:25.736 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.736 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.736 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.736 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.736 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.736 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.737 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [], 2024-04-12T16:55:25.737 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [], 2024-04-12T16:55:25.737 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:55:25.737 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:55:25.737 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:55:25.737 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:55:25.738 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.738 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:55:25.738 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:55:25.738 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:55:25.738 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.738 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:55:25.739 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:55:25.739 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:55:25.739 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.739 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.739 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.739 INFO:tasks.workunit.client.0.smithi137.stdout: "peer": "4(2)", 2024-04-12T16:55:25.739 INFO:tasks.workunit.client.0.smithi137.stdout: "pgid": "2.0s2", 2024-04-12T16:55:25.740 INFO:tasks.workunit.client.0.smithi137.stdout: "shared": "2", 2024-04-12T16:55:25.740 INFO:tasks.workunit.client.0.smithi137.stdout: "last_update": "48'101", 2024-04-12T16:55:25.740 INFO:tasks.workunit.client.0.smithi137.stdout: "last_complete": "48'101", 2024-04-12T16:55:25.740 INFO:tasks.workunit.client.0.smithi137.stdout: "log_tail": "0'0", 2024-04-12T16:55:25.740 INFO:tasks.workunit.client.0.smithi137.stdout: "last_user_version": 101, 2024-04-12T16:55:25.740 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill": "MAX", 2024-04-12T16:55:25.740 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [], 2024-04-12T16:55:25.741 INFO:tasks.workunit.client.0.smithi137.stdout: "history": { 2024-04-12T16:55:25.741 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_created": 41, 2024-04-12T16:55:25.741 INFO:tasks.workunit.client.0.smithi137.stdout: "epoch_pool_created": 41, 2024-04-12T16:55:25.741 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.741 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_started": 51, 2024-04-12T16:55:25.741 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.741 INFO:tasks.workunit.client.0.smithi137.stdout: "last_interval_clean": 41, 2024-04-12T16:55:25.742 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_split": 0, 2024-04-12T16:55:25.742 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_marked_full": 0, 2024-04-12T16:55:25.742 INFO:tasks.workunit.client.0.smithi137.stdout: "same_up_since": 51, 2024-04-12T16:55:25.742 INFO:tasks.workunit.client.0.smithi137.stdout: "same_interval_since": 51, 2024-04-12T16:55:25.742 INFO:tasks.workunit.client.0.smithi137.stdout: "same_primary_since": 41, 2024-04-12T16:55:25.742 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.742 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.743 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.743 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.743 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.743 INFO:tasks.workunit.client.0.smithi137.stdout: "prior_readable_until_ub": 0 2024-04-12T16:55:25.743 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.743 INFO:tasks.workunit.client.0.smithi137.stdout: "stats": { 2024-04-12T16:55:25.743 INFO:tasks.workunit.client.0.smithi137.stdout: "version": "48'101", 2024-04-12T16:55:25.744 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_seq": 131, 2024-04-12T16:55:25.744 INFO:tasks.workunit.client.0.smithi137.stdout: "reported_epoch": 48, 2024-04-12T16:55:25.744 INFO:tasks.workunit.client.0.smithi137.stdout: "state": "active+undersized+degraded", 2024-04-12T16:55:25.744 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fresh": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.744 INFO:tasks.workunit.client.0.smithi137.stdout: "last_change": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.744 INFO:tasks.workunit.client.0.smithi137.stdout: "last_active": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.744 INFO:tasks.workunit.client.0.smithi137.stdout: "last_peered": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.745 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean": "2024-04-12T16:54:22.825374+0000", 2024-04-12T16:55:25.745 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_active": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.745 INFO:tasks.workunit.client.0.smithi137.stdout: "last_became_peered": "2024-04-12T16:54:25.172177+0000", 2024-04-12T16:55:25.745 INFO:tasks.workunit.client.0.smithi137.stdout: "last_unstale": "2024-04-12T16:55:01.074097+0000", 2024-04-12T16:55:25.745 INFO:tasks.workunit.client.0.smithi137.stdout: "last_undegraded": "2024-04-12T16:54:24.716957+0000", 2024-04-12T16:55:25.745 INFO:tasks.workunit.client.0.smithi137.stdout: "last_fullsized": "2024-04-12T16:54:24.716708+0000", 2024-04-12T16:55:25.745 INFO:tasks.workunit.client.0.smithi137.stdout: "mapping_epoch": 51, 2024-04-12T16:55:25.746 INFO:tasks.workunit.client.0.smithi137.stdout: "log_start": "0'0", 2024-04-12T16:55:25.746 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_start": "0'0", 2024-04-12T16:55:25.746 INFO:tasks.workunit.client.0.smithi137.stdout: "created": 41, 2024-04-12T16:55:25.746 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_clean": 42, 2024-04-12T16:55:25.746 INFO:tasks.workunit.client.0.smithi137.stdout: "parent": "0.0", 2024-04-12T16:55:25.746 INFO:tasks.workunit.client.0.smithi137.stdout: "parent_split_bits": 0, 2024-04-12T16:55:25.746 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub": "0'0", 2024-04-12T16:55:25.747 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.747 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub": "0'0", 2024-04-12T16:55:25.747 INFO:tasks.workunit.client.0.smithi137.stdout: "last_deep_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.747 INFO:tasks.workunit.client.0.smithi137.stdout: "last_clean_scrub_stamp": "2024-04-12T16:54:13.844588+0000", 2024-04-12T16:55:25.747 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_scrubbed": 0, 2024-04-12T16:55:25.747 INFO:tasks.workunit.client.0.smithi137.stdout: "log_size": 101, 2024-04-12T16:55:25.747 INFO:tasks.workunit.client.0.smithi137.stdout: "log_dups_size": 0, 2024-04-12T16:55:25.747 INFO:tasks.workunit.client.0.smithi137.stdout: "ondisk_log_size": 101, 2024-04-12T16:55:25.748 INFO:tasks.workunit.client.0.smithi137.stdout: "stats_invalid": false, 2024-04-12T16:55:25.748 INFO:tasks.workunit.client.0.smithi137.stdout: "dirty_stats_invalid": false, 2024-04-12T16:55:25.748 INFO:tasks.workunit.client.0.smithi137.stdout: "omap_stats_invalid": false, 2024-04-12T16:55:25.748 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_stats_invalid": false, 2024-04-12T16:55:25.748 INFO:tasks.workunit.client.0.smithi137.stdout: "hitset_bytes_stats_invalid": false, 2024-04-12T16:55:25.748 INFO:tasks.workunit.client.0.smithi137.stdout: "pin_stats_invalid": false, 2024-04-12T16:55:25.748 INFO:tasks.workunit.client.0.smithi137.stdout: "manifest_stats_invalid": false, 2024-04-12T16:55:25.749 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrimq_len": 0, 2024-04-12T16:55:25.749 INFO:tasks.workunit.client.0.smithi137.stdout: "last_scrub_duration": 0, 2024-04-12T16:55:25.749 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-04-13T20:48:00.998122+0000", 2024-04-12T16:55:25.749 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_duration": 0, 2024-04-12T16:55:25.749 INFO:tasks.workunit.client.0.smithi137.stdout: "objects_trimmed": 0, 2024-04-12T16:55:25.749 INFO:tasks.workunit.client.0.smithi137.stdout: "snaptrim_duration": 0, 2024-04-12T16:55:25.749 INFO:tasks.workunit.client.0.smithi137.stdout: "stat_sum": { 2024-04-12T16:55:25.749 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes": 413696, 2024-04-12T16:55:25.750 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects": 101, 2024-04-12T16:55:25.750 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_clones": 0, 2024-04-12T16:55:25.750 INFO:tasks.workunit.client.0.smithi137.stdout: "num_object_copies": 505, 2024-04-12T16:55:25.750 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing_on_primary": 0, 2024-04-12T16:55:25.750 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_missing": 0, 2024-04-12T16:55:25.750 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_degraded": 101, 2024-04-12T16:55:25.750 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_misplaced": 0, 2024-04-12T16:55:25.751 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_unfound": 0, 2024-04-12T16:55:25.751 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_dirty": 101, 2024-04-12T16:55:25.751 INFO:tasks.workunit.client.0.smithi137.stdout: "num_whiteouts": 0, 2024-04-12T16:55:25.751 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read": 0, 2024-04-12T16:55:25.751 INFO:tasks.workunit.client.0.smithi137.stdout: "num_read_kb": 0, 2024-04-12T16:55:25.752 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write": 101, 2024-04-12T16:55:25.752 INFO:tasks.workunit.client.0.smithi137.stdout: "num_write_kb": 404, 2024-04-12T16:55:25.752 INFO:tasks.workunit.client.0.smithi137.stdout: "num_scrub_errors": 0, 2024-04-12T16:55:25.752 INFO:tasks.workunit.client.0.smithi137.stdout: "num_shallow_scrub_errors": 0, 2024-04-12T16:55:25.752 INFO:tasks.workunit.client.0.smithi137.stdout: "num_deep_scrub_errors": 0, 2024-04-12T16:55:25.752 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_recovered": 0, 2024-04-12T16:55:25.753 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_recovered": 0, 2024-04-12T16:55:25.753 INFO:tasks.workunit.client.0.smithi137.stdout: "num_keys_recovered": 0, 2024-04-12T16:55:25.753 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_omap": 0, 2024-04-12T16:55:25.753 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_hit_set_archive": 0, 2024-04-12T16:55:25.753 INFO:tasks.workunit.client.0.smithi137.stdout: "num_bytes_hit_set_archive": 0, 2024-04-12T16:55:25.753 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_kb": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_kb": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_promote": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_high": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_flush_mode_low": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_some": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_evict_mode_full": 0, 2024-04-12T16:55:25.754 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_pinned": 0, 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: "num_legacy_snapsets": 0, 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: "num_large_omap_objects": 0, 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_manifest": 0, 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_bytes": 0, 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: "num_omap_keys": 0, 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: "num_objects_repaired": 0 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: "up": [ 2024-04-12T16:55:25.755 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.756 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.756 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.756 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.756 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.756 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.756 INFO:tasks.workunit.client.0.smithi137.stdout: "acting": [ 2024-04-12T16:55:25.756 INFO:tasks.workunit.client.0.smithi137.stdout: 3, 2024-04-12T16:55:25.756 INFO:tasks.workunit.client.0.smithi137.stdout: 1, 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: 4, 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: 0, 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: 2 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: "avail_no_missing": [ 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: "3(0)", 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: "0(3)", 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: "1(1)", 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: "4(2)" 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: "object_location_counts": [ 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-04-12T16:55:25.757 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": 101 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: "blocked_by": [], 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: "up_primary": 3, 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: "acting_primary": 3, 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: "purged_snaps": [] 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: "empty": 0, 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: "dne": 0, 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: "incomplete": 0, 2024-04-12T16:55:25.758 INFO:tasks.workunit.client.0.smithi137.stdout: "last_epoch_started": 53, 2024-04-12T16:55:25.759 INFO:tasks.workunit.client.0.smithi137.stdout: "hit_set_history": { 2024-04-12T16:55:25.759 INFO:tasks.workunit.client.0.smithi137.stdout: "current_last_update": "0'0", 2024-04-12T16:55:25.759 INFO:tasks.workunit.client.0.smithi137.stdout: "history": [] 2024-04-12T16:55:25.759 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.759 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.759 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.759 INFO:tasks.workunit.client.0.smithi137.stdout: "recovery_state": [ 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "Started/Primary/Active", 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: "enter_time": "2024-04-12T16:55:10.080231+0000", 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: "might_have_unfound": [ 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "0(3)", 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "already probed" 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.760 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "1(1)", 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "already probed" 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "2(4)", 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "already probed" 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: "osd": "4(2)", 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: "status": "already probed" 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: "recovery_progress": { 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: "backfill_targets": [], 2024-04-12T16:55:25.761 INFO:tasks.workunit.client.0.smithi137.stdout: "waiting_on_backfill": [], 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "last_backfill_started": "MIN", 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "backfill_info": { 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "begin": "MIN", 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "end": "MIN", 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "objects": [] 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "peer_backfill_info": [], 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "backfills_in_flight": [], 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "recovering": [], 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "pg_backend": { 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "recovery_ops": [], 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: "read_ops": [] 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.762 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: { 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "name": "Started", 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "enter_time": "2024-04-12T16:55:09.199614+0000" 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: } 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: ], 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "scrubber": { 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "active": false, 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "must_scrub": false, 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "must_deep_scrub": false, 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "must_repair": false, 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "need_auto": false, 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "scrub_reg_stamp": "2024-04-13T20:48:00.998122+0000", 2024-04-12T16:55:25.763 INFO:tasks.workunit.client.0.smithi137.stdout: "schedule": "no scrub is scheduled" 2024-04-12T16:55:25.764 INFO:tasks.workunit.client.0.smithi137.stdout: }, 2024-04-12T16:55:25.764 INFO:tasks.workunit.client.0.smithi137.stdout: "agent_state": {} 2024-04-12T16:55:25.764 INFO:tasks.workunit.client.0.smithi137.stdout:} 2024-04-12T16:55:25.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: grep -q obj75 2024-04-12T16:55:25.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:55:25.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:55:25.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-04-12T16:55:26.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: check=true 2024-04-12T16:55:26.023 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:667: TEST_ec_recovery_unfound: test true == true 2024-04-12T16:55:26.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-04-12T16:55:26.023 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-04-12T16:55:26.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: check=0 2024-04-12T16:55:26.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: test 0 == 0 2024-04-12T16:55:26.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-04-12T16:55:31.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: test 124 = 124 2024-04-12T16:55:31.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:676: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-12T16:55:31.495 INFO:tasks.workunit.client.0.smithi137.stderr:pg has no unfound objects 2024-04-12T16:55:31.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:678: TEST_ec_recovery_unfound: wait_for_clean 2024-04-12T16:55:31.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T16:55:31.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T16:55:31.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T16:55:31.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T16:55:31.520 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T16:55:31.520 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T16:55:31.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T16:55:31.520 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T16:55:31.521 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T16:55:31.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T16:55:31.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T16:55:31.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T16:55:31.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T16:55:31.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T16:55:31.645 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T16:55:32.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T16:55:32.075 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T16:55:32.075 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T16:55:32.075 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T16:55:32.075 INFO:tasks.workunit.client.0.smithi137.stderr:4' 2024-04-12T16:55:32.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T16:55:32.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:55:32.076 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T16:55:32.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803842 2024-04-12T16:55:32.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803842 2024-04-12T16:55:32.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842' 2024-04-12T16:55:32.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:55:32.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T16:55:32.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607610 2024-04-12T16:55:32.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607610 2024-04-12T16:55:32.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-51539607610' 2024-04-12T16:55:32.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:55:32.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T16:55:32.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364807 2024-04-12T16:55:32.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364807 2024-04-12T16:55:32.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-51539607610 2-214748364807' 2024-04-12T16:55:32.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:55:32.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T16:55:32.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117030 2024-04-12T16:55:32.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117030 2024-04-12T16:55:32.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-51539607610 2-214748364807 3-115964117030' 2024-04-12T16:55:32.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T16:55:32.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T16:55:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789979 2024-04-12T16:55:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789979 2024-04-12T16:55:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-51539607610 2-214748364807 3-115964117030 4-158913789979' 2024-04-12T16:55:33.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:55:33.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803842 2024-04-12T16:55:33.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:55:33.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T16:55:33.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803842 2024-04-12T16:55:33.051 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:55:33.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803842 2024-04-12T16:55:33.053 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803842 2024-04-12T16:55:33.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-04-12T16:55:33.053 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:55:33.489 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803840 -lt 25769803842 2024-04-12T16:55:33.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T16:55:34.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T16:55:34.491 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T16:55:34.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803842 -lt 25769803842 2024-04-12T16:55:34.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:55:34.917 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607610 2024-04-12T16:55:34.918 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:55:34.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T16:55:34.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607610 2024-04-12T16:55:34.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:55:34.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607610 2024-04-12T16:55:34.922 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 51539607610 2024-04-12T16:55:34.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607610' 2024-04-12T16:55:34.923 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T16:55:35.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607610 -lt 51539607610 2024-04-12T16:55:35.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:55:35.309 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-214748364807 2024-04-12T16:55:35.309 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:55:35.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T16:55:35.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-214748364807 2024-04-12T16:55:35.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:55:35.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364807 2024-04-12T16:55:35.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 214748364807' 2024-04-12T16:55:35.312 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 214748364807 2024-04-12T16:55:35.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T16:55:35.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364808 -lt 214748364807 2024-04-12T16:55:35.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:55:35.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117030 2024-04-12T16:55:35.749 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:55:35.750 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T16:55:35.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117030 2024-04-12T16:55:35.751 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:55:35.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117030 2024-04-12T16:55:35.754 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 115964117030 2024-04-12T16:55:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117030' 2024-04-12T16:55:35.754 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T16:55:36.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117030 -lt 115964117030 2024-04-12T16:55:36.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T16:55:36.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-158913789979 2024-04-12T16:55:36.188 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T16:55:36.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T16:55:36.190 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-158913789979 2024-04-12T16:55:36.191 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T16:55:36.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789979 2024-04-12T16:55:36.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 158913789979' 2024-04-12T16:55:36.192 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 158913789979 2024-04-12T16:55:36.193 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T16:55:36.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789979 -lt 158913789979 2024-04-12T16:55:36.585 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T16:55:36.586 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:55:36.586 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:55:37.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T16:55:37.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T16:55:37.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T16:55:37.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T16:55:37.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T16:55:37.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T16:55:37.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T16:55:37.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T16:55:37.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T16:55:37.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T16:55:37.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T16:55:37.514 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T16:55:38.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T16:55:38.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T16:55:38.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T16:55:38.111 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: seq 1 100 2024-04-12T16:55:38.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-04-12T16:55:38.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-04-12T16:55:38.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-04-12T16:55:38.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-04-12T16:55:38.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.288 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-04-12T16:55:38.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-04-12T16:55:38.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-04-12T16:55:38.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.377 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-04-12T16:55:38.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-04-12T16:55:38.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-04-12T16:55:38.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-04-12T16:55:38.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-04-12T16:55:38.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-04-12T16:55:38.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-04-12T16:55:38.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-04-12T16:55:38.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-04-12T16:55:38.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-04-12T16:55:38.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.670 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.671 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-04-12T16:55:38.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.699 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-04-12T16:55:38.700 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-04-12T16:55:38.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-04-12T16:55:38.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-04-12T16:55:38.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.816 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-04-12T16:55:38.817 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-04-12T16:55:38.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.873 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-04-12T16:55:38.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-04-12T16:55:38.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.930 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-04-12T16:55:38.931 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.959 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-04-12T16:55:38.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-04-12T16:55:38.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:38.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:38.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-04-12T16:55:38.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-04-12T16:55:39.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-04-12T16:55:39.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-04-12T16:55:39.075 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-04-12T16:55:39.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-04-12T16:55:39.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-04-12T16:55:39.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-04-12T16:55:39.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-04-12T16:55:39.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-04-12T16:55:39.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-04-12T16:55:39.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-04-12T16:55:39.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-04-12T16:55:39.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-04-12T16:55:39.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-04-12T16:55:39.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-04-12T16:55:39.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-04-12T16:55:39.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-04-12T16:55:39.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-04-12T16:55:39.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-04-12T16:55:39.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-04-12T16:55:39.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-04-12T16:55:39.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.653 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-04-12T16:55:39.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-04-12T16:55:39.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-04-12T16:55:39.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-04-12T16:55:39.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-04-12T16:55:39.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.799 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-04-12T16:55:39.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-04-12T16:55:39.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.856 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-04-12T16:55:39.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-04-12T16:55:39.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-04-12T16:55:39.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-04-12T16:55:39.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-04-12T16:55:39.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:39.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:39.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-04-12T16:55:39.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-04-12T16:55:40.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-04-12T16:55:40.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.132 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-04-12T16:55:40.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-04-12T16:55:40.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-04-12T16:55:40.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-04-12T16:55:40.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-04-12T16:55:40.249 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-04-12T16:55:40.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-04-12T16:55:40.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-04-12T16:55:40.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-04-12T16:55:40.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-04-12T16:55:40.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-04-12T16:55:40.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.447 INFO:tasks.workunit.client.0.smithi137.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-04-12T16:55:40.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-04-12T16:55:40.449 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-04-12T16:55:40.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-04-12T16:55:40.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-04-12T16:55:40.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-04-12T16:55:40.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-04-12T16:55:40.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-04-12T16:55:40.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-04-12T16:55:40.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-04-12T16:55:40.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-04-12T16:55:40.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-04-12T16:55:40.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-04-12T16:55:40.766 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-04-12T16:55:40.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-04-12T16:55:40.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.851 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-04-12T16:55:40.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-04-12T16:55:40.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-04-12T16:55:40.911 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-04-12T16:55:40.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-04-12T16:55:40.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-04-12T16:55:40.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:40.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:40.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-04-12T16:55:40.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-04-12T16:55:41.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:41.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:41.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-04-12T16:55:41.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-04-12T16:55:41.055 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:41.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:41.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-04-12T16:55:41.056 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-04-12T16:55:41.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:41.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:41.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-04-12T16:55:41.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-04-12T16:55:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:41.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:41.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-04-12T16:55:41.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-04-12T16:55:41.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:41.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-12T16:55:41.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-04-12T16:55:41.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-04-12T16:55:41.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:41.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:691: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-04-12T16:55:41.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-04-12T16:55:41.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T16:55:41.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T16:55:41.724 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T16:55:41.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-04-12T16:55:42.924 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile myprofile does not exist 2024-04-12T16:55:42.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-04-12T16:55:42.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:55:42.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:55:42.947 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:55:42.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:55:42.947 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:55:42.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:55:42.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:55:42.948 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:55:43.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:55:43.263 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:55:43.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:55:43.264 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:55:43.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:55:43.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:55:43.266 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:55:43.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:55:43.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:55:43.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:55:43.268 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:55:43.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:55:43.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:55:43.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:55:43.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:55:43.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:43.318 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:43.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:55:43.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T16:55:43.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T16:55:43.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T16:55:43.323 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T16:55:43.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T16:55:43.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T16:55:43.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T16:55:43.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T16:55:43.325 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T16:55:43.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:55:43.326 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T16:55:43.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T16:55:43.327 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T16:55:43.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T16:55:43.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T16:55:43.329 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T16:55:43.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T16:55:43.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:43.331 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:43.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T16:55:43.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T16:55:43.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T16:55:43.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T16:55:43.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T16:55:43.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:43.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:43.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T16:55:43.335 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T16:55:43.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T16:55:43.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T16:55:43.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:55:43.841 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:55:43.842 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:55:43.842 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:43.842 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:43.842 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:55:43.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T16:55:44.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T16:55:44.102 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T16:55:44.102 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:55:44.102 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:55:44.102 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T16:55:44.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T16:55:44.103 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:55:44.103 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:55:44.103 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:55:44.103 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:55:44.103 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:44.104 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:44.104 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:55:44.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:55:44.104 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T16:55:44.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T16:55:44.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T16:55:44.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T16:55:44.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T16:55:44.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T16:55:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T16:55:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:55:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:55:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:55:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:44.217 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:55:44.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T16:55:44.218 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T16:55:44.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T16:55:44.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T16:55:44.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T16:55:44.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T16:55:44.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T16:55:44.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T16:55:44.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T16:55:45.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T16:55:45.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:55:45.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:55:45.332 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:55:45.332 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:45.332 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:45.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:55:45.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:55:45.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T16:55:45.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T16:55:45.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T16:55:45.738 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T16:55:45.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T16:55:46.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T16:55:46.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T16:55:46.759 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T16:55:46.760 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T16:55:46.760 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:46.760 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:46.760 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T16:55:46.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T16:55:46.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T16:55:46.860 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T16:55:46.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T16:55:46.874 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:55:43.887+0000 7f797eae3c80 0 load: jerasure load: lrc 2024-04-12T16:55:46.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-04-12T16:55:46.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-04-12T16:55:46.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:391: TEST_ec_single_recovery_error: local objname=myobject 2024-04-12T16:55:46.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:393: TEST_ec_single_recovery_error: setup_osds 7 2024-04-12T16:55:46.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-04-12T16:55:46.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T16:55:46.876 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-04-12T16:55:46.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-04-12T16:55:46.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:55:46.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T16:55:46.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:55:46.881 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:55:46.882 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:55:46.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:55:46.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:55:46.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:55:46.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T16:55:46.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:55:46.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc1eb27e-097e-499a-8962-e7db847014af 2024-04-12T16:55:46.884 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 dc1eb27e-097e-499a-8962-e7db847014af 2024-04-12T16:55:46.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dc1eb27e-097e-499a-8962-e7db847014af' 2024-04-12T16:55:46.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:55:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSZxlm3A/VNRAAeNgUnNSi9qAihJGxlcvONA== 2024-04-12T16:55:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSZxlm3A/VNRAAeNgUnNSi9qAihJGxlcvONA=="}' 2024-04-12T16:55:46.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc1eb27e-097e-499a-8962-e7db847014af -i td/test-erasure-eio/0/new.json 2024-04-12T16:55:47.256 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:55:47.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T16:55:47.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSZxlm3A/VNRAAeNgUnNSi9qAihJGxlcvONA== --osd-uuid dc1eb27e-097e-499a-8962-e7db847014af 2024-04-12T16:55:47.305 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:47.303+0000 7fe8daba37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:47.341 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:47.343+0000 7fe8daba37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:47.343 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:47.343+0000 7fe8daba37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:47.343 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:47.343+0000 7fe8daba37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:55:47.343 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:47.343+0000 7fe8daba37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:55:47.343 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:47.343+0000 7fe8daba37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T16:55:47.382 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:47.383+0000 7fe8daba37c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T16:55:55.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T16:55:55.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:55:55.477 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T16:55:55.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T16:55:55.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:55:56.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T16:55:56.050 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T16:55:56.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:55:56.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:55:56.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:55:56.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:55:56.081 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:56.079+0000 7f3bd2e877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:56.083 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:56.083+0000 7f3bd2e877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:56.084 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:56.083+0000 7f3bd2e877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:55:56.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T16:55:56.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:55:56.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T16:55:56.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:55:56.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:55:56.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:55:56.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:55:56.448 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:55:56.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:55:56.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:55:56.632 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:56.631+0000 7f3bd2e877c0 -1 Falling back to public interface 2024-04-12T16:55:56.912 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:55:57.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:55:57.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:55:57.914 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:55:57.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:55:57.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:55:57.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:55:58.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:55:59.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:55:59.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:55:59.338 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:55:59.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:55:59.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:55:59.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:55:59.444 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:55:59.443+0000 7f3bd2e877c0 -1 osd.0 0 log_to_monitors true 2024-04-12T16:55:59.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:00.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:00.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:00.831 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:56:00.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:56:00.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:00.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:01.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:02.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:02.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:02.259 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:56:02.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:56:02.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:02.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:02.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:03.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:03.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:03.828 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:56:03.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:56:03.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:03.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:04.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:05.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:05.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:05.241 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:56:05.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:56:05.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:05.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:05.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:06.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:06.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:06.878 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:56:06.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:56:06.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:06.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:07.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:08.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:08.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:08.523 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:56:08.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:56:08.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:08.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:08.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:10.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:10.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:10.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:56:10.001 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:56:10.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:10.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:10.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:11.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:11.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:11.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:56:11.602 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:56:11.602 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:11.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:12.099 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:13.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:13.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:13.101 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:56:13.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:56:13.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:13.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:13.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:14.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:14.620 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:14.621 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:56:14.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:56:14.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:14.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:15.176 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:16.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:16.178 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:16.178 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:56:16.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:56:16.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:16.179 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:16.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:17.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:17.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:17.794 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:56:17.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:56:17.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:17.795 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:18.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:19.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:19.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:19.645 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:56:19.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:56:19.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:19.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:20.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:21.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:21.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:21.634 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:56:21.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:56:21.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:21.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:22.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:23.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:23.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:23.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:56:23.643 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:56:23.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:23.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:24.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:25.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:25.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:25.466 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:56:25.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:56:25.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:25.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:26.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:27.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:27.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:27.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:56:27.266 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:56:27.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:27.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:28.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:29.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:29.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:29.375 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:56:29.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:56:29.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:29.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:30.001 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:31.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:31.002 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:31.003 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:56:31.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:56:31.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:31.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T16:56:31.527 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3571787972,v1:127.0.0.1:6803/3571787972] [v2:127.0.0.1:6804/3571787972,v1:127.0.0.1:6805/3571787972] exists,up dc1eb27e-097e-499a-8962-e7db847014af 2024-04-12T16:56:31.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:56:31.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:56:31.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:56:31.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:56:31.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:56:31.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:56:31.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:56:31.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:56:31.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:56:31.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:56:31.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:56:31.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:56:31.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:56:31.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T16:56:31.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:56:31.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa6b667c-3cbe-492a-b46c-919ea2154ac2 2024-04-12T16:56:31.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 aa6b667c-3cbe-492a-b46c-919ea2154ac2' 2024-04-12T16:56:31.535 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 aa6b667c-3cbe-492a-b46c-919ea2154ac2 2024-04-12T16:56:31.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:56:31.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/Zxlm0DUpIRAAf9Iy80Ra3+CgaAlGLW1Iaw== 2024-04-12T16:56:31.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/Zxlm0DUpIRAAf9Iy80Ra3+CgaAlGLW1Iaw=="}' 2024-04-12T16:56:31.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa6b667c-3cbe-492a-b46c-919ea2154ac2 -i td/test-erasure-eio/1/new.json 2024-04-12T16:56:32.190 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:56:32.207 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T16:56:32.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/Zxlm0DUpIRAAf9Iy80Ra3+CgaAlGLW1Iaw== --osd-uuid aa6b667c-3cbe-492a-b46c-919ea2154ac2 2024-04-12T16:56:32.242 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:32.243+0000 7faa0f0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:56:32.245 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:32.243+0000 7faa0f0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:56:32.246 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:32.247+0000 7faa0f0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:56:32.246 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:32.247+0000 7faa0f0537c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:56:32.246 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:32.247+0000 7faa0f0537c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:56:32.246 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:32.247+0000 7faa0f0537c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T16:56:32.315 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:32.315+0000 7faa0f0537c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T16:56:40.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T16:56:40.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:56:40.260 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T16:56:40.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T16:56:40.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:56:40.841 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T16:56:40.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T16:56:40.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:56:40.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:56:40.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:56:40.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:56:40.872 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:40.871+0000 7ffa8991c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:56:40.874 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:40.875+0000 7ffa8991c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:56:40.875 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:40.875+0000 7ffa8991c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:56:41.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T16:56:41.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:56:41.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T16:56:41.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:56:41.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:56:41.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:41.239 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:56:41.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:56:41.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:41.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:41.415 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:41.415+0000 7ffa8991c7c0 -1 Falling back to public interface 2024-04-12T16:56:41.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:42.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:42.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:42.636 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:56:42.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:56:42.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:42.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:43.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:44.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:44.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:44.066 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:56:44.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:56:44.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:44.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:44.287 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:56:44.287+0000 7ffa8991c7c0 -1 osd.1 0 log_to_monitors true 2024-04-12T16:56:44.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:45.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:45.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:45.532 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:56:45.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:56:45.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:45.532 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:46.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:47.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:47.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:47.139 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:56:47.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:56:47.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:47.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:47.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:48.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:48.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:48.596 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:56:48.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:56:48.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:48.596 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:49.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:50.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:50.089 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:50.089 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:56:50.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:56:50.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:50.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:50.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:51.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:51.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:51.663 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:56:51.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:56:51.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:51.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:52.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:53.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:53.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:53.166 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:56:53.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:56:53.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:53.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:53.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:54.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:54.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:54.683 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:56:54.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:56:54.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:54.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:55.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:56.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:56.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:56.270 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:56:56.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:56:56.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:56.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:56.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:57.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:57.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:57.780 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:56:57.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:56:57.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:57.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:58.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:56:59.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:56:59.375 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:56:59.376 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:56:59.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:56:59.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:56:59.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:56:59.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:00.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:00.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:00.896 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:57:00.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:57:00.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:00.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:01.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:02.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:02.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:02.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:57:02.555 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:57:02.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:02.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:03.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:04.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:04.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:04.062 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:57:04.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:57:04.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:04.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:04.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:05.675 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:57:05.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:57:05.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:05.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:06.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:07.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:07.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:07.543 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:57:07.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:57:07.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:07.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:08.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:09.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:09.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:09.650 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:57:09.651 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:57:09.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:09.651 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:10.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:11.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:11.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:11.398 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:57:11.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:57:11.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:11.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:12.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:13.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:13.229 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:13.229 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:57:13.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:57:13.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:13.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:14.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:15.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:15.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:15.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:57:15.185 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:57:15.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:15.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:15.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:16.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:16.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:16.973 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:57:16.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:57:16.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:16.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T16:57:17.657 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/185008094,v1:127.0.0.1:6811/185008094] [v2:127.0.0.1:6812/185008094,v1:127.0.0.1:6813/185008094] exists,up aa6b667c-3cbe-492a-b46c-919ea2154ac2 2024-04-12T16:57:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:57:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:57:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:57:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:57:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T16:57:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:57:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:57:17.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:57:17.662 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:57:17.663 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:57:17.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:57:17.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:57:17.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:57:17.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:57:17.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:57:17.664 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T16:57:17.665 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:57:17.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0ec165c-1ae8-427f-ac87-0a66f5ef8e18 2024-04-12T16:57:17.665 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c0ec165c-1ae8-427f-ac87-0a66f5ef8e18' 2024-04-12T16:57:17.665 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 c0ec165c-1ae8-427f-ac87-0a66f5ef8e18 2024-04-12T16:57:17.666 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:57:17.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtZxlmMdLbKBAAHyjS3dwRoVp5pxh+Ks7yyw== 2024-04-12T16:57:17.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtZxlmMdLbKBAAHyjS3dwRoVp5pxh+Ks7yyw=="}' 2024-04-12T16:57:17.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0ec165c-1ae8-427f-ac87-0a66f5ef8e18 -i td/test-erasure-eio/2/new.json 2024-04-12T16:57:18.627 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:57:18.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T16:57:18.648 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtZxlmMdLbKBAAHyjS3dwRoVp5pxh+Ks7yyw== --osd-uuid c0ec165c-1ae8-427f-ac87-0a66f5ef8e18 2024-04-12T16:57:18.683 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:18.683+0000 7fa4a13797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:57:18.741 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:18.739+0000 7fa4a13797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:57:18.743 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:18.743+0000 7fa4a13797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:57:18.743 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:18.743+0000 7fa4a13797c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:57:18.743 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:18.743+0000 7fa4a13797c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:57:18.743 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:18.743+0000 7fa4a13797c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T16:57:18.849 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:18.847+0000 7fa4a13797c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T16:57:26.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T16:57:26.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:57:26.919 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T16:57:26.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T16:57:26.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:57:27.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T16:57:27.492 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T16:57:27.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:57:27.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:57:27.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:57:27.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:57:27.518 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:27.515+0000 7f7197bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:57:27.520 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:27.519+0000 7f7197bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:57:27.521 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:27.519+0000 7f7197bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:57:27.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T16:57:27.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:57:27.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T16:57:27.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:57:27.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:57:27.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:27.902 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:57:27.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:57:27.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:27.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:28.068 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:28.067+0000 7f7197bf77c0 -1 Falling back to public interface 2024-04-12T16:57:28.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:29.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:29.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:29.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:57:29.293 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:57:29.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:29.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:29.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:30.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:30.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:30.715 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:57:30.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:57:30.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:30.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:31.019 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:57:31.019+0000 7f7197bf77c0 -1 osd.2 0 log_to_monitors true 2024-04-12T16:57:31.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:32.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:32.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:32.142 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:57:32.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:57:32.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:32.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:32.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:33.808 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:33.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:33.809 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:57:33.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:57:33.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:33.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:34.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:35.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:35.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:35.498 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:57:35.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:57:35.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:35.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:35.987 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:36.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:36.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:36.989 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:57:36.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:57:36.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:36.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:37.592 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:38.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:38.594 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:38.595 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:57:38.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:57:38.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:38.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:39.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:40.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:40.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:40.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:57:40.150 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:57:40.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:40.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:40.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:41.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:41.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:41.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:57:41.769 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:57:41.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:41.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:42.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:43.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:43.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:43.481 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:57:43.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:57:43.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:43.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:43.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:44.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:44.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:44.953 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:57:44.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:57:44.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:44.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:45.731 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:46.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:46.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:46.734 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:57:46.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:57:46.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:46.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:47.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:48.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:48.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:48.399 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:57:48.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:57:48.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:48.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:48.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:49.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:49.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:49.979 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:57:49.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:57:49.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:49.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:51.111 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:52.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:52.113 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:52.113 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:57:52.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:57:52.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:52.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:53.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:54.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:54.161 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:54.162 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:57:54.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:57:54.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:54.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:54.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:55.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:55.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:55.910 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:57:55.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:57:55.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:55.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:56.972 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:57.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:57.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:57.975 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:57:57.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:57:57.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:57.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:57:58.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:57:59.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:57:59.746 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:57:59.746 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:57:59.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:57:59.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:57:59.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:58:00.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:01.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:01.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:01.720 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:58:01.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:58:01.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:01.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:58:02.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:03.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:03.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:03.338 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:58:03.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:58:03.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:03.339 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T16:58:04.313 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/421499185,v1:127.0.0.1:6819/421499185] [v2:127.0.0.1:6820/421499185,v1:127.0.0.1:6821/421499185] exists,up c0ec165c-1ae8-427f-ac87-0a66f5ef8e18 2024-04-12T16:58:04.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:58:04.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:58:04.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:58:04.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:58:04.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T16:58:04.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:58:04.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:58:04.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:58:04.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:58:04.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:58:04.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:58:04.317 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:58:04.317 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:58:04.317 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:58:04.317 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:58:04.319 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T16:58:04.320 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:58:04.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58fa9184-b0a2-4946-a3f8-6d0c729a863f 2024-04-12T16:58:04.320 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 58fa9184-b0a2-4946-a3f8-6d0c729a863f 2024-04-12T16:58:04.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 58fa9184-b0a2-4946-a3f8-6d0c729a863f' 2024-04-12T16:58:04.321 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:58:04.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcaBlm/JlAFBAAiFYVeQEftnZjaMirOFYaog== 2024-04-12T16:58:04.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcaBlm/JlAFBAAiFYVeQEftnZjaMirOFYaog=="}' 2024-04-12T16:58:04.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58fa9184-b0a2-4946-a3f8-6d0c729a863f -i td/test-erasure-eio/3/new.json 2024-04-12T16:58:05.699 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:58:05.720 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T16:58:05.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcaBlm/JlAFBAAiFYVeQEftnZjaMirOFYaog== --osd-uuid 58fa9184-b0a2-4946-a3f8-6d0c729a863f 2024-04-12T16:58:05.755 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:05.755+0000 7f6d03bcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:05.787 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:05.787+0000 7f6d03bcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:05.788 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:05.787+0000 7f6d03bcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:05.789 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:05.787+0000 7f6d03bcb7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:58:05.789 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:05.787+0000 7f6d03bcb7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:58:05.789 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:05.787+0000 7f6d03bcb7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T16:58:05.929 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:05.927+0000 7f6d03bcb7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T16:58:13.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T16:58:13.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:58:13.921 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T16:58:13.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T16:58:13.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:58:14.507 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T16:58:14.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T16:58:14.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:58:14.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:58:14.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:58:14.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:58:14.538 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:14.535+0000 7f11103077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:14.541 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:14.539+0000 7f11103077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:14.542 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:14.539+0000 7f11103077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:14.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T16:58:14.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:58:14.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T16:58:14.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:58:14.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:58:14.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:14.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:58:14.910 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:58:14.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:14.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:15.084 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:15.083+0000 7f11103077c0 -1 Falling back to public interface 2024-04-12T16:58:15.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:16.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:16.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:16.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:58:16.365 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:58:16.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:16.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:16.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:17.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:17.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:17.791 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:58:17.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:58:17.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:17.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:17.956 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:17.955+0000 7f11103077c0 -1 osd.3 0 log_to_monitors true 2024-04-12T16:58:18.239 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:19.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:19.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:19.241 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:58:19.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:58:19.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:19.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:19.902 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:20.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:20.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:20.905 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:58:20.905 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:58:20.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:20.906 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:21.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:22.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:22.468 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:22.468 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:58:22.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:58:22.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:22.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:23.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:24.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:24.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:24.127 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:58:24.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:58:24.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:24.128 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:24.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:25.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:25.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:25.604 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:58:25.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:58:25.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:25.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:26.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:27.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:27.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:27.403 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:58:27.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:58:27.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:27.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:27.869 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:28.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:28.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:28.871 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:58:28.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:58:28.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:28.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:29.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:30.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:30.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:30.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:58:30.497 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:58:30.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:30.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:30.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:31.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:31.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:31.952 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:58:31.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:58:31.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:31.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:32.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:33.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:33.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:33.661 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:58:33.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:58:33.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:33.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:34.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:35.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:35.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:35.336 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:58:35.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:58:35.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:35.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:35.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:36.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:36.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:36.791 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:58:36.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:58:36.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:36.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:37.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:38.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:38.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:38.453 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:58:38.453 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:58:38.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:38.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:39.400 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:40.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:40.402 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:40.402 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:58:40.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:58:40.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:40.403 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:41.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:42.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:42.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:42.561 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:58:42.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:58:42.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:42.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:43.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:44.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:44.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:44.916 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:58:44.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:58:44.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:44.917 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:45.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:46.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:46.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:46.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:58:46.759 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:58:46.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:46.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:47.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:48.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:48.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:48.679 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:58:48.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:58:48.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:48.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:49.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:58:50.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:58:50.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:58:50.992 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:58:50.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:58:50.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:58:50.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T16:58:51.460 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4235812228,v1:127.0.0.1:6827/4235812228] [v2:127.0.0.1:6828/4235812228,v1:127.0.0.1:6829/4235812228] exists,up 58fa9184-b0a2-4946-a3f8-6d0c729a863f 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:58:51.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:58:51.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:58:51.466 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-04-12T16:58:51.467 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:58:51.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=06cea280-b768-40cd-be68-7e4d2cbd8396 2024-04-12T16:58:51.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 06cea280-b768-40cd-be68-7e4d2cbd8396' 2024-04-12T16:58:51.467 INFO:tasks.workunit.client.0.smithi137.stdout:add osd4 06cea280-b768-40cd-be68-7e4d2cbd8396 2024-04-12T16:58:51.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:58:51.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLaBlmbSDDHBAARDbncjnJmHAqN3jGTENlEw== 2024-04-12T16:58:51.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLaBlmbSDDHBAARDbncjnJmHAqN3jGTENlEw=="}' 2024-04-12T16:58:51.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 06cea280-b768-40cd-be68-7e4d2cbd8396 -i td/test-erasure-eio/4/new.json 2024-04-12T16:58:52.887 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:58:52.914 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-04-12T16:58:52.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLaBlmbSDDHBAARDbncjnJmHAqN3jGTENlEw== --osd-uuid 06cea280-b768-40cd-be68-7e4d2cbd8396 2024-04-12T16:58:52.950 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:52.947+0000 7f55aee5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:52.981 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:52.979+0000 7f55aee5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:52.983 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:52.983+0000 7f55aee5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:58:52.984 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:52.983+0000 7f55aee5e7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:58:52.984 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:52.983+0000 7f55aee5e7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:58:52.984 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:52.983+0000 7f55aee5e7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-04-12T16:58:53.239 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:58:53.239+0000 7f55aee5e7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-04-12T16:59:01.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-04-12T16:59:01.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:59:01.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-12T16:59:01.487 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd4 key to auth repository 2024-04-12T16:59:01.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:59:02.056 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.4 2024-04-12T16:59:02.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-12T16:59:02.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:59:02.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:59:02.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:59:02.057 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:59:02.088 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:02.087+0000 7f39eaa077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:02.090 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:02.087+0000 7f39eaa077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:02.091 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:02.091+0000 7f39eaa077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:02.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-12T16:59:02.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:59:02.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-12T16:59:02.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:59:02.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:59:02.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:02.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:59:02.478 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:59:02.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:02.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:02.644 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:02.643+0000 7f39eaa077c0 -1 Falling back to public interface 2024-04-12T16:59:02.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:03.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:03.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:03.877 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:59:03.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:59:03.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:03.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:04.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:05.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:05.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:05.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:59:05.301 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:59:05.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:05.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:05.457 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:05.455+0000 7f39eaa077c0 -1 osd.4 0 log_to_monitors true 2024-04-12T16:59:05.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:06.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:06.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:06.784 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:59:06.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:59:06.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:06.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:07.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:08.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:08.539 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:59:08.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:08.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:59:08.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:08.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:09.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:10.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:10.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:10.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:59:10.431 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:59:10.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:10.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:11.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:12.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:12.076 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:12.076 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T16:59:12.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T16:59:12.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:12.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:12.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:13.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:13.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:13.677 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T16:59:13.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T16:59:13.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:13.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:14.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:15.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:15.139 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:15.139 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T16:59:15.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T16:59:15.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:15.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:15.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:16.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:16.796 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:16.796 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T16:59:16.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T16:59:16.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:16.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:17.635 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:18.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:18.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:18.637 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T16:59:18.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T16:59:18.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:18.637 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:19.068 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:20.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:20.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:20.071 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T16:59:20.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T16:59:20.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:20.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:20.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:21.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:21.696 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:21.696 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T16:59:21.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T16:59:21.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:21.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:22.182 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:23.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:23.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:23.184 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T16:59:23.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T16:59:23.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:23.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:23.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:24.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:24.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:24.726 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T16:59:24.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T16:59:24.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:24.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:25.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:26.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:26.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:26.408 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T16:59:26.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T16:59:26.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:26.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:27.266 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:28.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:28.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:28.269 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T16:59:28.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T16:59:28.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:28.269 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:29.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:30.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:30.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:30.584 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T16:59:30.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T16:59:30.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:30.584 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:31.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:32.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:32.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:32.187 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T16:59:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T16:59:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:32.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:33.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:34.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:34.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:34.169 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T16:59:34.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T16:59:34.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:34.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:35.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:36.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:36.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:36.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T16:59:36.029 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T16:59:36.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:36.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:36.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:37.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:37.903 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:37.903 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T16:59:37.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T16:59:37.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:37.904 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:38.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:39.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:39.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:39.641 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T16:59:39.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T16:59:39.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:39.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-12T16:59:40.244 INFO:tasks.workunit.client.0.smithi137.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2135650060,v1:127.0.0.1:6835/2135650060] [v2:127.0.0.1:6836/2135650060,v1:127.0.0.1:6837/2135650060] exists,up 06cea280-b768-40cd-be68-7e4d2cbd8396 2024-04-12T16:59:40.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T16:59:40.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T16:59:40.246 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-04-12T16:59:40.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-04-12T16:59:40.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T16:59:40.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T16:59:40.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T16:59:40.248 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T16:59:40.248 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T16:59:40.248 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T16:59:40.249 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T16:59:40.250 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T16:59:40.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T16:59:40.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T16:59:40.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T16:59:40.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-04-12T16:59:40.251 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T16:59:40.251 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a8d14a58-098e-4958-9bb4-7652b485ae83 2024-04-12T16:59:40.251 INFO:tasks.workunit.client.0.smithi137.stdout:add osd5 a8d14a58-098e-4958-9bb4-7652b485ae83 2024-04-12T16:59:40.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a8d14a58-098e-4958-9bb4-7652b485ae83' 2024-04-12T16:59:40.252 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T16:59:40.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8aBlmu5pIEBAAbfblzdMhV48dqHpidHYcUQ== 2024-04-12T16:59:40.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8aBlmu5pIEBAAbfblzdMhV48dqHpidHYcUQ=="}' 2024-04-12T16:59:40.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a8d14a58-098e-4958-9bb4-7652b485ae83 -i td/test-erasure-eio/5/new.json 2024-04-12T16:59:41.374 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:59:41.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-04-12T16:59:41.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8aBlmu5pIEBAAbfblzdMhV48dqHpidHYcUQ== --osd-uuid a8d14a58-098e-4958-9bb4-7652b485ae83 2024-04-12T16:59:41.427 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:41.423+0000 7f0bd91a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:41.429 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:41.427+0000 7f0bd91a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:41.430 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:41.427+0000 7f0bd91a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:41.431 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:41.427+0000 7f0bd91a57c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:59:41.431 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:41.427+0000 7f0bd91a57c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:59:41.431 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:41.427+0000 7f0bd91a57c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-04-12T16:59:41.758 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:41.755+0000 7f0bd91a57c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-04-12T16:59:50.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-04-12T16:59:50.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T16:59:50.338 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd5 key to auth repository 2024-04-12T16:59:50.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-12T16:59:50.338 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T16:59:50.978 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.5 2024-04-12T16:59:50.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-12T16:59:50.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T16:59:50.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T16:59:50.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T16:59:50.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T16:59:51.009 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:51.007+0000 7f8b392bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:51.050 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:51.047+0000 7f8b392bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:51.051 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:51.051+0000 7f8b392bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T16:59:51.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-12T16:59:51.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T16:59:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-12T16:59:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T16:59:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T16:59:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T16:59:51.444 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T16:59:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:59:51.638 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:51.635+0000 7f8b392bc7c0 -1 Falling back to public interface 2024-04-12T16:59:51.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:52.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:52.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:52.921 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T16:59:52.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T16:59:52.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:52.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:59:53.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:54.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:54.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:54.336 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T16:59:54.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T16:59:54.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:54.337 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:59:54.603 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T16:59:54.599+0000 7f8b392bc7c0 -1 osd.5 0 log_to_monitors true 2024-04-12T16:59:54.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:55.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:55.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:55.882 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T16:59:55.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T16:59:55.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:55.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:59:56.954 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:57.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:57.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:57.956 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T16:59:57.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T16:59:57.956 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:57.957 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T16:59:58.794 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T16:59:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T16:59:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T16:59:59.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T16:59:59.797 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T16:59:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T16:59:59.798 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:00.221 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:01.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:01.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:01.223 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:00:01.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:00:01.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:01.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:01.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:02.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:02.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:02.824 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:00:02.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:00:02.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:02.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:03.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:04.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:04.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:04.267 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:00:04.267 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:00:04.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:04.268 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:04.923 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:05.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:05.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:05.925 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:00:05.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:00:05.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:05.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:06.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:07.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:07.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:07.366 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:00:07.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:00:07.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:07.367 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:07.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:08.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:08.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:08.974 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:00:08.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:00:08.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:08.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:09.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:10.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:10.509 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:10.510 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:00:10.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:00:10.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:10.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:11.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:12.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:12.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:12.053 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:00:12.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:00:12.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:12.054 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:12.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:13.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:13.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:13.523 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:00:13.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:00:13.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:13.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:14.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:15.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:15.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:15.103 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:00:15.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:00:15.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:15.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:15.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:16.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:16.927 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:16.927 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:00:16.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:00:16.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:16.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:17.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:18.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:18.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:18.727 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:00:18.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:00:18.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:18.728 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:19.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:20.827 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:20.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:20.828 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:00:20.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:00:20.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:20.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:22.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:23.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:23.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:23.085 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:00:23.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:00:23.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:23.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:24.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:25.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:25.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:25.167 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:00:25.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:00:25.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:25.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:25.973 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:26.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:26.975 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:26.976 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:00:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:00:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:26.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:27.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:28.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:28.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:28.862 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T17:00:28.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T17:00:28.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:28.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-12T17:00:29.558 INFO:tasks.workunit.client.0.smithi137.stdout:osd.5 up in weight 1 up_from 47 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3736557259,v1:127.0.0.1:6843/3736557259] [v2:127.0.0.1:6844/3736557259,v1:127.0.0.1:6845/3736557259] exists,up a8d14a58-098e-4958-9bb4-7652b485ae83 2024-04-12T17:00:29.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:00:29.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:00:29.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:00:29.564 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:00:29.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:00:29.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:00:29.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-04-12T17:00:29.565 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:00:29.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3b2dfa0d-8d90-4c06-861c-c1ec18f85931 2024-04-12T17:00:29.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 3b2dfa0d-8d90-4c06-861c-c1ec18f85931' 2024-04-12T17:00:29.566 INFO:tasks.workunit.client.0.smithi137.stdout:add osd6 3b2dfa0d-8d90-4c06-861c-c1ec18f85931 2024-04-12T17:00:29.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:00:29.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtaBlmDATiIhAAJNtpCFAqo5ledYwsCnV/6A== 2024-04-12T17:00:29.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtaBlmDATiIhAAJNtpCFAqo5ledYwsCnV/6A=="}' 2024-04-12T17:00:29.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3b2dfa0d-8d90-4c06-861c-c1ec18f85931 -i td/test-erasure-eio/6/new.json 2024-04-12T17:00:30.573 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:00:30.591 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-04-12T17:00:30.593 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtaBlmDATiIhAAJNtpCFAqo5ledYwsCnV/6A== --osd-uuid 3b2dfa0d-8d90-4c06-861c-c1ec18f85931 2024-04-12T17:00:30.622 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:30.619+0000 7f33fb68b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:00:30.695 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:30.691+0000 7f33fb68b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:00:30.696 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:30.695+0000 7f33fb68b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:00:30.697 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:30.695+0000 7f33fb68b7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T17:00:30.697 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:30.695+0000 7f33fb68b7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T17:00:30.697 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:30.695+0000 7f33fb68b7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-04-12T17:00:31.013 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:31.011+0000 7f33fb68b7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-04-12T17:00:39.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-04-12T17:00:39.450 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:00:39.451 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd6 key to auth repository 2024-04-12T17:00:39.451 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-12T17:00:39.452 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:00:40.003 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-12T17:00:40.003 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.6 2024-04-12T17:00:40.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:00:40.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:00:40.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:00:40.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:00:40.034 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:40.031+0000 7f4821eeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:00:40.036 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:40.035+0000 7f4821eeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:00:40.037 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:40.035+0000 7f4821eeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:00:40.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-12T17:00:40.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:00:40.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-12T17:00:40.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:00:40.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:00:40.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:40.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:00:40.421 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:00:40.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:40.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:40.584 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:40.583+0000 7f4821eeb7c0 -1 Falling back to public interface 2024-04-12T17:00:40.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:41.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:41.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:41.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:00:41.831 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:00:41.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:41.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:42.289 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:43.290 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:43.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:43.291 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:00:43.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:00:43.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:43.292 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:43.472 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:00:43.471+0000 7f4821eeb7c0 -1 osd.6 0 log_to_monitors true 2024-04-12T17:00:43.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:44.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:44.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:44.781 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:00:44.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:00:44.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:44.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:45.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:46.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:46.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:46.539 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:00:46.539 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:00:46.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:46.540 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:47.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:48.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:48.121 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:48.121 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:00:48.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:00:48.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:48.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:48.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:49.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:49.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:49.557 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:00:49.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:00:49.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:49.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:50.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:51.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:51.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:51.149 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:00:51.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:00:51.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:51.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:51.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:52.791 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:52.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:52.792 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:00:52.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:00:52.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:52.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:53.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:54.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:54.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:54.393 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:00:54.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:00:54.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:54.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:54.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:55.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:55.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:55.885 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:00:55.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:00:55.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:55.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:56.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:57.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:57.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:57.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:00:57.324 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:00:57.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:57.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:57.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:00:58.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:00:58.882 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:00:58.882 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:00:58.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:00:58.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:00:58.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:00:59.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:00.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:00.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:00.478 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:01:00.478 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:01:00.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:00.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:01.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:02.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:02.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:02.157 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:01:02.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:01:02.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:02.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:02.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:03.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:03.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:03.707 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:01:03.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:01:03.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:03.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:04.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:05.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:05.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:05.507 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:01:05.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:01:05.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:05.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:06.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:07.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:07.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:07.348 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:01:07.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:01:07.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:07.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:08.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:09.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:09.236 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:09.237 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:01:09.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:01:09.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:09.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:10.127 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:11.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:11.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:11.129 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:01:11.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:01:11.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:11.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:11.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:12.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:12.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:12.758 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:01:12.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:01:12.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:12.758 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:13.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:14.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:14.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:14.439 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:01:14.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:01:14.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:14.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:15.997 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:16.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:16.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:16.999 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T17:01:17.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T17:01:17.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:17.000 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-12T17:01:17.495 INFO:tasks.workunit.client.0.smithi137.stdout:osd.6 up in weight 1 up_from 57 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3312014531,v1:127.0.0.1:6851/3312014531] [v2:127.0.0.1:6852/3312014531,v1:127.0.0.1:6853/3312014531] exists,up 3b2dfa0d-8d90-4c06-861c-c1ec18f85931 2024-04-12T17:01:17.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:01:17.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:01:17.496 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:01:17.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T17:01:17.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T17:01:17.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T17:01:17.497 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:01:17.497 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:17.497 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:17.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T17:01:17.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T17:01:17.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T17:01:17.609 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T17:01:17.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T17:01:17.627 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T16:55:56.639+0000 7f3bd2e877c0 0 load: jerasure load: lrc 2024-04-12T17:01:17.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-04-12T17:01:17.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:396: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-04-12T17:01:17.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T17:01:17.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T17:01:17.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-04-12T17:01:17.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T17:01:17.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-04-12T17:01:17.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T17:01:17.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-04-12T17:01:19.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T17:01:19.227 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T17:01:20.815 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T17:01:20.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T17:01:21.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T17:01:21.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:01:21.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:01:21.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:01:21.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:01:21.838 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:01:21.839 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:01:21.839 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:01:21.839 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:01:21.839 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:01:21.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:01:21.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:01:21.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:01:21.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:01:21.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:01:21.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:01:22.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:01:22.396 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:01:22.396 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:01:22.396 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T17:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T17:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T17:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:22.397 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:01:22.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803843 2024-04-12T17:01:22.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803843 2024-04-12T17:01:22.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843' 2024-04-12T17:01:22.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:22.686 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:01:22.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607609 2024-04-12T17:01:22.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607609 2024-04-12T17:01:22.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-51539607609' 2024-04-12T17:01:22.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:22.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:01:23.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378669 2024-04-12T17:01:23.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378669 2024-04-12T17:01:23.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-51539607609 2-81604378669' 2024-04-12T17:01:23.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:23.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:01:23.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051622 2024-04-12T17:01:23.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051622 2024-04-12T17:01:23.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-51539607609 2-81604378669 3-124554051622' 2024-04-12T17:01:23.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:23.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T17:01:23.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757280 2024-04-12T17:01:23.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757280 2024-04-12T17:01:23.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-51539607609 2-81604378669 3-124554051622 4-163208757280' 2024-04-12T17:01:23.546 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:23.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T17:01:23.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462929 2024-04-12T17:01:23.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462929 2024-04-12T17:01:23.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-51539607609 2-81604378669 3-124554051622 4-163208757280 5-201863462929' 2024-04-12T17:01:23.705 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:23.705 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T17:01:23.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135885 2024-04-12T17:01:23.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135885 2024-04-12T17:01:23.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-51539607609 2-81604378669 3-124554051622 4-163208757280 5-201863462929 6-244813135885' 2024-04-12T17:01:23.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:23.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803843 2024-04-12T17:01:23.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:23.862 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:01:23.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803843 2024-04-12T17:01:23.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:23.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803843 2024-04-12T17:01:23.865 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803843 2024-04-12T17:01:23.865 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803843' 2024-04-12T17:01:23.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:01:24.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803843 -lt 25769803843 2024-04-12T17:01:24.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:24.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607609 2024-04-12T17:01:24.308 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:24.309 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:01:24.310 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607609 2024-04-12T17:01:24.310 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:24.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607609 2024-04-12T17:01:24.312 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 51539607609 2024-04-12T17:01:24.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607609' 2024-04-12T17:01:24.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:01:24.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607609 -lt 51539607609 2024-04-12T17:01:24.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:24.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378669 2024-04-12T17:01:24.761 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:24.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:01:24.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378669 2024-04-12T17:01:24.763 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:24.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378669 2024-04-12T17:01:24.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378669' 2024-04-12T17:01:24.765 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378669 2024-04-12T17:01:24.765 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:01:25.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378670 -lt 81604378669 2024-04-12T17:01:25.211 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:25.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-124554051622 2024-04-12T17:01:25.212 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:25.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:01:25.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-124554051622 2024-04-12T17:01:25.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:25.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051622 2024-04-12T17:01:25.216 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 124554051622 2024-04-12T17:01:25.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 124554051622' 2024-04-12T17:01:25.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:01:25.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051622 -lt 124554051622 2024-04-12T17:01:25.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:25.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757280 2024-04-12T17:01:25.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:25.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T17:01:25.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757280 2024-04-12T17:01:25.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:25.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757280 2024-04-12T17:01:25.614 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 163208757280 2024-04-12T17:01:25.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757280' 2024-04-12T17:01:25.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T17:01:26.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757280 -lt 163208757280 2024-04-12T17:01:26.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:26.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-201863462929 2024-04-12T17:01:26.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:26.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T17:01:26.046 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-201863462929 2024-04-12T17:01:26.046 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:26.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462929 2024-04-12T17:01:26.047 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 201863462929 2024-04-12T17:01:26.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 201863462929' 2024-04-12T17:01:26.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T17:01:26.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462930 -lt 201863462929 2024-04-12T17:01:26.465 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:26.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:26.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-244813135885 2024-04-12T17:01:26.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-12T17:01:26.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-244813135885 2024-04-12T17:01:26.468 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:26.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135885 2024-04-12T17:01:26.469 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.6 seq 244813135885 2024-04-12T17:01:26.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 244813135885' 2024-04-12T17:01:26.470 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-12T17:01:26.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135885 -lt 244813135885 2024-04-12T17:01:26.918 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:01:26.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:01:26.919 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:01:27.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:01:27.446 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:01:27.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:01:27.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:01:27.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:01:27.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:01:27.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:01:27.447 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:01:27.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:01:27.922 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:01:27.922 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:01:27.922 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:01:28.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T17:01:28.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:01:28.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T17:01:28.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:01:28.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T17:01:28.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:399: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-04-12T17:01:28.744 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-12T17:01:28.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-04-12T17:01:28.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-12T17:01:28.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-12T17:01:28.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-12T17:01:28.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-04-12T17:01:28.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:01:28.745 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T17:01:28.745 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:01:28.746 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T17:01:29.215 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-04-12T17:01:29.216 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-04-12T17:01:29.217 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-04-12T17:01:29.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-12T17:01:29.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-12T17:01:29.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-12T17:01:29.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-12T17:01:29.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-12T17:01:29.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-12T17:01:29.219 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-12T17:01:29.219 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-12T17:01:29.220 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T17:01:29.220 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T17:01:29.220 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:01:29.221 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:29.221 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:29.221 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T17:01:29.221 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79474/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-12T17:01:29.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-12T17:01:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-12T17:01:29.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-04-12T17:01:29.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-12T17:01:29.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-12T17:01:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-12T17:01:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:01:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:29.344 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:29.344 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.3.asok 2024-04-12T17:01:29.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-12T17:01:29.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-04-12T17:01:29.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-04-12T17:01:29.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:01:29.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-12T17:01:29.466 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:01:29.466 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-04-12T17:01:29.895 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:01:29.895 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T17:01:29.895 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:01:29.896 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:01:29.896 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T17:01:29.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-04-12T17:01:29.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2024-04-12T17:01:29.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local -a initial_osds 2024-04-12T17:01:29.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:402: TEST_ec_single_recovery_error: local last_osd=6 2024-04-12T17:01:29.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-04-12T17:01:29.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:01:29.897 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:01:29.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:01:29.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:01:29.897 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:01:30.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:01:30.204 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd down 6 2024-04-12T17:01:31.364 INFO:tasks.workunit.client.0.smithi137.stderr:osd.6 is already down. 2024-04-12T17:01:31.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:406: TEST_ec_single_recovery_error: ceph osd out 6 2024-04-12T17:01:32.140 INFO:tasks.workunit.client.0.smithi137.stderr:osd.6 is already out. 2024-04-12T17:01:32.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:409: TEST_ec_single_recovery_error: wait_for_clean 2024-04-12T17:01:32.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:01:32.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:01:32.159 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:01:32.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:01:32.160 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:01:32.160 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:01:32.161 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:01:32.161 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:01:32.161 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:01:32.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:01:32.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:01:32.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:01:32.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:01:32.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:01:32.272 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:01:33.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr:4 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr:5 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr:6' 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:33.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:01:33.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803847 2024-04-12T17:01:33.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803847 2024-04-12T17:01:33.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847' 2024-04-12T17:01:33.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:33.419 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:01:33.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607613 2024-04-12T17:01:33.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607613 2024-04-12T17:01:33.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-51539607613' 2024-04-12T17:01:33.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:33.587 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:01:33.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378673 2024-04-12T17:01:33.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378673 2024-04-12T17:01:33.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-51539607613 2-81604378673' 2024-04-12T17:01:33.825 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:33.825 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:01:34.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051626 2024-04-12T17:01:34.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051626 2024-04-12T17:01:34.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-51539607613 2-81604378673 3-124554051626' 2024-04-12T17:01:34.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:34.071 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-12T17:01:34.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757284 2024-04-12T17:01:34.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757284 2024-04-12T17:01:34.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-51539607613 2-81604378673 3-124554051626 4-163208757284' 2024-04-12T17:01:34.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:34.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-12T17:01:34.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462934 2024-04-12T17:01:34.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462934 2024-04-12T17:01:34.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803847 1-51539607613 2-81604378673 3-124554051626 4-163208757284 5-201863462934' 2024-04-12T17:01:34.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:01:34.469 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-12T17:01:34.806 INFO:tasks.workunit.client.0.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-12T17:01:34.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-04-12T17:01:34.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-04-12T17:01:34.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-04-12T17:01:34.809 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:34.809 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803847 2024-04-12T17:01:34.810 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:34.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:01:34.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803847 2024-04-12T17:01:34.814 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:34.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803847 2024-04-12T17:01:34.815 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803847 2024-04-12T17:01:34.815 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803847' 2024-04-12T17:01:34.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:01:35.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803847 -lt 25769803847 2024-04-12T17:01:35.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:35.368 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607613 2024-04-12T17:01:35.369 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:35.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:01:35.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607613 2024-04-12T17:01:35.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607613 2024-04-12T17:01:35.373 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 51539607613 2024-04-12T17:01:35.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607613' 2024-04-12T17:01:35.373 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:01:35.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607613 -lt 51539607613 2024-04-12T17:01:35.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:35.787 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378673 2024-04-12T17:01:35.788 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:35.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:01:35.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378673 2024-04-12T17:01:35.791 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:35.792 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378673 2024-04-12T17:01:35.792 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378673 2024-04-12T17:01:35.793 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378673' 2024-04-12T17:01:35.793 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:01:36.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378674 -lt 81604378673 2024-04-12T17:01:36.197 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:36.198 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-124554051626 2024-04-12T17:01:36.198 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:36.199 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:01:36.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-124554051626 2024-04-12T17:01:36.200 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:36.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051626 2024-04-12T17:01:36.203 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 124554051626 2024-04-12T17:01:36.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 124554051626' 2024-04-12T17:01:36.203 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:01:36.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051626 -lt 124554051626 2024-04-12T17:01:36.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:36.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757284 2024-04-12T17:01:36.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:36.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-12T17:01:36.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757284 2024-04-12T17:01:36.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:36.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757284 2024-04-12T17:01:36.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757284' 2024-04-12T17:01:36.613 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.4 seq 163208757284 2024-04-12T17:01:36.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-12T17:01:37.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757285 -lt 163208757284 2024-04-12T17:01:37.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:01:37.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-201863462934 2024-04-12T17:01:37.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:01:37.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-12T17:01:37.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-201863462934 2024-04-12T17:01:37.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:01:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462934 2024-04-12T17:01:37.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 201863462934' 2024-04-12T17:01:37.054 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.5 seq 201863462934 2024-04-12T17:01:37.054 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-12T17:01:37.515 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462934 -lt 201863462934 2024-04-12T17:01:37.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:01:37.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:01:37.516 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:01:38.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:01:38.106 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:01:38.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:01:38.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:01:38.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:01:38.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:01:38.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:01:38.107 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:01:38.507 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:01:38.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:01:38.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:01:38.508 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:01:39.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:01:39.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:01:39.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:01:39.059 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-04-12T17:01:39.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:01:39.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:01:39.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-04-12T17:01:39.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T17:01:39.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T17:01:39.060 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-04-12T17:01:39.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T17:01:39.094 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T17:01:39.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-04-12T17:01:39.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T17:01:39.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T17:01:39.670 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T17:01:39.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-04-12T17:01:40.695 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile myprofile does not exist 2024-04-12T17:01:40.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-04-12T17:01:40.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T17:01:40.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T17:01:40.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T17:01:40.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:01:40.713 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:01:40.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:01:40.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:01:40.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:01:41.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:01:41.028 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T17:01:41.029 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T17:01:41.030 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T17:01:41.031 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T17:01:41.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T17:01:41.032 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T17:01:41.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:01:41.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T17:01:41.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T17:01:41.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:01:41.035 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T17:01:41.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T17:01:41.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T17:01:41.079 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T17:01:41.080 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:41.080 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:41.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T17:01:41.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T17:01:41.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T17:01:41.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:01:41.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:01:41.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:01:41.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T17:01:41.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T17:01:41.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T17:01:41.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T17:01:41.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T17:01:41.087 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T17:01:41.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:01:41.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T17:01:41.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T17:01:41.089 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:01:41.090 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T17:01:41.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T17:01:41.091 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T17:01:41.092 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T17:01:41.092 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:41.093 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:41.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T17:01:41.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T17:01:41.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T17:01:41.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T17:01:41.095 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T17:01:41.095 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:41.095 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T17:01:41.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T17:01:41.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T17:01:41.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T17:01:41.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:01:41.614 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:01:41.614 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:01:41.615 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:41.615 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:41.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:01:41.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T17:01:41.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T17:01:41.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T17:01:41.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T17:01:41.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T17:01:41.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T17:01:41.851 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T17:01:41.851 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T17:01:41.851 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T17:01:41.851 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T17:01:41.852 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:01:41.852 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:41.852 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:41.852 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T17:01:41.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T17:01:41.852 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T17:01:41.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T17:01:41.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T17:01:41.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T17:01:41.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T17:01:41.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T17:01:41.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T17:01:41.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T17:01:41.965 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T17:01:41.965 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:01:41.965 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:41.966 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:41.966 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T17:01:41.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T17:01:41.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T17:01:42.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T17:01:42.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T17:01:42.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T17:01:42.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T17:01:42.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T17:01:42.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T17:01:42.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T17:01:43.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T17:01:43.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:01:43.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:01:43.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:01:43.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:43.048 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:43.048 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:01:43.049 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T17:01:43.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T17:01:43.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T17:01:43.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T17:01:43.445 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T17:01:43.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T17:01:44.465 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T17:01:44.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T17:01:44.466 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T17:01:44.467 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:01:44.467 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:44.467 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:44.467 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T17:01:44.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T17:01:44.467 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T17:01:44.586 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T17:01:44.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T17:01:44.603 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T17:01:41.655+0000 7f2d2cea4c80 0 load: jerasure load: lrc 2024-04-12T17:01:44.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-04-12T17:01:44.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-04-12T17:01:44.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-04-12T17:01:44.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-04-12T17:01:44.604 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T17:01:44.605 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-04-12T17:01:44.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-04-12T17:01:44.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:01:44.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T17:01:44.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:01:44.610 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:01:44.611 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:01:44.612 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T17:01:44.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:01:44.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1e59ab4-60e4-4013-a270-fd4f4cc5a2fb 2024-04-12T17:01:44.614 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 f1e59ab4-60e4-4013-a270-fd4f4cc5a2fb 2024-04-12T17:01:44.614 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f1e59ab4-60e4-4013-a270-fd4f4cc5a2fb' 2024-04-12T17:01:44.615 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:01:44.629 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4aBlmIjN3JRAAg9QxZY8TgOm0MO9MJRnQlQ== 2024-04-12T17:01:44.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4aBlmIjN3JRAAg9QxZY8TgOm0MO9MJRnQlQ=="}' 2024-04-12T17:01:44.630 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1e59ab4-60e4-4013-a270-fd4f4cc5a2fb -i td/test-erasure-eio/0/new.json 2024-04-12T17:01:44.996 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:01:45.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T17:01:45.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4aBlmIjN3JRAAg9QxZY8TgOm0MO9MJRnQlQ== --osd-uuid f1e59ab4-60e4-4013-a270-fd4f4cc5a2fb 2024-04-12T17:01:45.047 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:45.043+0000 7f003580e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:01:45.082 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:45.079+0000 7f003580e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:01:45.083 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:45.079+0000 7f003580e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:01:45.084 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:45.083+0000 7f003580e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T17:01:45.084 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:45.083+0000 7f003580e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T17:01:45.084 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:45.083+0000 7f003580e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T17:01:45.121 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:45.119+0000 7f003580e7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T17:01:53.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T17:01:53.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:01:53.243 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T17:01:53.244 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T17:01:53.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:01:53.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T17:01:53.831 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T17:01:53.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:01:53.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:01:53.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:01:53.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:01:53.857 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:53.855+0000 7f56c2c707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:01:53.863 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:53.859+0000 7f56c2c707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:01:53.863 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:53.859+0000 7f56c2c707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:01:54.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T17:01:54.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:01:54.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T17:01:54.264 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:01:54.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:01:54.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:01:54.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:54.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:01:54.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:01:54.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:54.417 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:54.415+0000 7f56c2c707c0 -1 Falling back to public interface 2024-04-12T17:01:54.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:55.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:55.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:55.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:01:55.718 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:01:55.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:55.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:01:56.122 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:57.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:57.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:57.124 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:01:57.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:01:57.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:57.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:01:57.184 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:01:57.183+0000 7f56c2c707c0 -1 osd.0 0 log_to_monitors true 2024-04-12T17:01:57.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:01:58.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:01:58.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:01:58.569 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:01:58.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:01:58.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:01:58.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:01:59.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:00.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:00.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:00.259 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:02:00.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:02:00.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:00.260 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:00.715 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:01.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:01.717 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:01.717 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:02:01.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:02:01.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:01.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:02.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:03.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:03.263 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:03.263 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:02:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:02:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:03.264 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:03.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:04.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:04.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:04.762 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:02:04.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:02:04.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:04.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:05.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:06.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:06.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:06.417 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:02:06.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:02:06.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:06.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:06.907 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:07.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:07.909 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:07.909 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:02:07.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:02:07.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:07.910 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:08.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:09.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:09.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:09.375 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:02:09.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:02:09.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:09.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:09.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:10.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:10.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:10.832 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:02:10.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:02:10.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:10.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:11.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:12.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:12.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:12.302 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:02:12.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:02:12.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:12.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:13.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:14.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:14.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:14.017 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:02:14.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:02:14.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:14.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:14.657 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:15.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:15.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:15.660 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:02:15.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:02:15.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:15.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:16.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:17.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:17.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:17.344 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:02:17.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:02:17.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:17.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:18.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:19.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:19.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:19.190 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:02:19.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:02:19.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:19.191 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:19.989 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:20.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:20.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:20.991 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:02:20.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:02:20.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:20.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:21.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:22.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:22.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:22.962 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:02:22.962 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:02:22.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:22.963 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:24.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:25.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:25.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:25.079 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:02:25.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:02:25.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:25.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:25.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:26.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:26.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:26.898 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:02:26.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:02:26.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:26.899 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:27.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:28.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:28.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:28.759 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:02:28.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:02:28.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:28.760 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:02:29.337 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3528688047,v1:127.0.0.1:6803/3528688047] [v2:127.0.0.1:6804/3528688047,v1:127.0.0.1:6805/3528688047] exists,up f1e59ab4-60e4-4013-a270-fd4f4cc5a2fb 2024-04-12T17:02:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:02:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:02:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:02:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:02:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T17:02:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:02:29.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:02:29.342 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:02:29.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:02:29.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:02:29.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:02:29.344 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T17:02:29.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:02:29.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0a91350-689e-430c-9299-6a4468635f7f 2024-04-12T17:02:29.345 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b0a91350-689e-430c-9299-6a4468635f7f' 2024-04-12T17:02:29.345 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 b0a91350-689e-430c-9299-6a4468635f7f 2024-04-12T17:02:29.345 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:02:29.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlaRlmXS+0FRAA2xSMhXqxj53UYuV25k9ROw== 2024-04-12T17:02:29.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlaRlmXS+0FRAA2xSMhXqxj53UYuV25k9ROw=="}' 2024-04-12T17:02:29.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0a91350-689e-430c-9299-6a4468635f7f -i td/test-erasure-eio/1/new.json 2024-04-12T17:02:30.139 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:02:30.155 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T17:02:30.156 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlaRlmXS+0FRAA2xSMhXqxj53UYuV25k9ROw== --osd-uuid b0a91350-689e-430c-9299-6a4468635f7f 2024-04-12T17:02:30.183 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:30.179+0000 7f26474517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:02:30.184 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:30.183+0000 7f26474517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:02:30.185 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:30.183+0000 7f26474517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:02:30.186 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:30.183+0000 7f26474517c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T17:02:30.186 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:30.183+0000 7f26474517c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T17:02:30.186 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:30.183+0000 7f26474517c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T17:02:30.263 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:30.259+0000 7f26474517c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T17:02:38.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T17:02:38.133 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:02:38.135 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T17:02:38.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T17:02:38.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:02:38.724 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T17:02:38.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T17:02:38.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:02:38.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:02:38.725 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:02:38.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:02:38.756 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:38.755+0000 7f630f7457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:02:38.758 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:38.755+0000 7f630f7457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:02:38.759 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:38.755+0000 7f630f7457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:02:39.129 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T17:02:39.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:02:39.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T17:02:39.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:02:39.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:02:39.130 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:39.130 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:02:39.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:02:39.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:39.131 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:39.310 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:39.307+0000 7f630f7457c0 -1 Falling back to public interface 2024-04-12T17:02:39.583 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:40.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:40.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:40.585 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:02:40.586 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:02:40.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:40.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:41.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:42.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:42.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:42.012 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:02:42.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:02:42.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:42.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:42.047 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:02:42.039+0000 7f630f7457c0 -1 osd.1 0 log_to_monitors true 2024-04-12T17:02:42.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:43.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:43.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:43.472 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:02:43.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:02:43.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:43.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:43.918 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:44.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:44.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:44.921 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:02:44.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:02:44.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:44.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:45.415 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:46.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:46.417 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:46.417 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:02:46.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:02:46.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:46.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:47.041 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:48.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:48.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:48.043 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:02:48.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:02:48.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:48.044 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:48.819 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:49.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:49.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:49.821 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:02:49.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:02:49.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:49.822 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:50.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:51.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:51.444 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:02:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:02:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:51.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:52.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:53.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:53.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:53.014 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:02:53.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:02:53.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:53.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:53.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:54.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:54.555 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:54.556 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:02:54.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:02:54.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:54.556 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:55.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:56.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:56.261 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:56.262 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:02:56.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:02:56.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:56.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:56.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:57.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:57.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:57.836 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:02:57.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:02:57.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:57.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:02:58.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:02:59.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:02:59.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:02:59.513 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:02:59.514 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:02:59.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:02:59.514 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:00.030 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:01.032 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:01.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:01.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:03:01.033 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:03:01.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:01.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:01.852 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:02.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:02.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:02.854 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:03:02.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:03:02.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:02.855 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:03.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:04.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:04.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:04.889 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:03:04.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:03:04.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:04.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:05.765 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:06.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:06.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:06.767 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:03:06.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:03:06.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:06.768 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:07.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:08.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:08.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:08.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:03:08.627 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:03:08.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:08.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:09.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:10.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:10.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:10.386 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:03:10.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:03:10.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:10.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:11.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:12.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:12.572 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:12.572 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:03:12.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:03:12.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:12.573 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:13.358 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:14.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:14.360 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:14.360 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:03:14.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:03:14.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:14.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:03:15.295 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3840924997,v1:127.0.0.1:6811/3840924997] [v2:127.0.0.1:6812/3840924997,v1:127.0.0.1:6813/3840924997] exists,up b0a91350-689e-430c-9299-6a4468635f7f 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T17:03:15.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:03:15.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:03:15.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:03:15.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:03:15.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T17:03:15.303 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:03:15.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43672208-1d55-4d7c-916f-79576787733e 2024-04-12T17:03:15.303 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 43672208-1d55-4d7c-916f-79576787733e 2024-04-12T17:03:15.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 43672208-1d55-4d7c-916f-79576787733e' 2024-04-12T17:03:15.304 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:03:15.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTaRlmww9fExAACXHZIZ7ebGYAVO+AsDuxbQ== 2024-04-12T17:03:15.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTaRlmww9fExAACXHZIZ7ebGYAVO+AsDuxbQ=="}' 2024-04-12T17:03:15.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43672208-1d55-4d7c-916f-79576787733e -i td/test-erasure-eio/2/new.json 2024-04-12T17:03:15.906 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:03:15.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T17:03:15.924 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTaRlmww9fExAACXHZIZ7ebGYAVO+AsDuxbQ== --osd-uuid 43672208-1d55-4d7c-916f-79576787733e 2024-04-12T17:03:15.961 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:15.955+0000 7f51444c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:03:15.963 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:15.959+0000 7f51444c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:03:15.963 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:15.959+0000 7f51444c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:03:15.964 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:15.959+0000 7f51444c87c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T17:03:15.964 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:15.959+0000 7f51444c87c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T17:03:15.964 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:15.959+0000 7f51444c87c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T17:03:16.068 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:16.063+0000 7f51444c87c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T17:03:24.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T17:03:24.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:03:24.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T17:03:24.281 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T17:03:24.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:03:24.848 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T17:03:24.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T17:03:24.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:03:24.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:03:24.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:03:24.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:03:24.881 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:24.879+0000 7febb85f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:03:24.883 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:24.879+0000 7febb85f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:03:24.884 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:24.883+0000 7febb85f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:03:25.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T17:03:25.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:03:25.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T17:03:25.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:03:25.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:03:25.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:25.242 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:03:25.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:03:25.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:25.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:25.426 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:25.423+0000 7febb85f67c0 -1 Falling back to public interface 2024-04-12T17:03:25.695 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:26.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:26.697 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:26.697 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:03:26.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:03:26.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:26.698 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:27.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:28.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:28.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:28.103 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:03:28.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:03:28.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:28.104 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:28.346 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:03:28.343+0000 7febb85f67c0 -1 osd.2 0 log_to_monitors true 2024-04-12T17:03:28.631 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:29.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:29.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:29.633 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:03:29.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:03:29.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:29.633 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:30.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:31.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:31.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:31.340 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:03:31.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:03:31.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:31.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:32.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:33.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:33.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:33.100 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:03:33.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:03:33.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:33.101 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:33.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:34.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:34.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:34.693 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:03:34.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:03:34.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:34.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:35.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:36.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:36.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:36.145 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:03:36.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:03:36.145 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:36.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:36.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:37.712 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:37.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:37.713 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:03:37.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:03:37.713 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:37.714 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:38.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:39.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:39.222 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:39.222 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:03:39.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:03:39.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:39.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:39.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:40.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:40.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:40.783 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:03:40.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:03:40.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:40.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:41.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:42.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:42.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:42.398 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:03:42.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:03:42.398 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:42.399 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:43.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:44.026 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:44.027 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:44.027 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:03:44.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:03:44.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:44.028 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:44.541 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:45.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:45.543 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:45.543 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:03:45.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:03:45.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:45.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:46.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:47.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:47.084 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:03:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:03:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:47.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:47.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:48.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:48.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:48.499 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:03:48.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:03:48.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:48.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:49.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:50.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:50.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:50.535 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:03:50.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:03:50.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:50.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:51.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:52.459 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:52.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:52.460 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:03:52.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:03:52.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:52.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:53.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:54.842 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:54.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:54.843 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:03:54.843 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:03:54.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:54.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:55.716 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:56.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:56.718 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:56.719 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:03:56.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:03:56.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:56.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:57.404 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:03:58.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:03:58.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:03:58.405 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:03:58.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:03:58.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:03:58.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:03:59.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:00.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:00.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:00.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:04:00.844 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:04:00.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:00.845 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:04:01.677 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1673417563,v1:127.0.0.1:6819/1673417563] [v2:127.0.0.1:6820/1673417563,v1:127.0.0.1:6821/1673417563] exists,up 43672208-1d55-4d7c-916f-79576787733e 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:04:01.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:04:01.682 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:04:01.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:04:01.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:04:01.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:04:01.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:04:01.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:04:01.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:04:01.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T17:04:01.684 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:04:01.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1dcf5d98-e067-47c2-b383-8e29d196a9a6 2024-04-12T17:04:01.685 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 1dcf5d98-e067-47c2-b383-8e29d196a9a6 2024-04-12T17:04:01.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1dcf5d98-e067-47c2-b383-8e29d196a9a6' 2024-04-12T17:04:01.685 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:04:01.708 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBaRlmKcgcKhAAPVdm0uXx/Hz2dy4qy/8/zg== 2024-04-12T17:04:01.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBaRlmKcgcKhAAPVdm0uXx/Hz2dy4qy/8/zg=="}' 2024-04-12T17:04:01.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1dcf5d98-e067-47c2-b383-8e29d196a9a6 -i td/test-erasure-eio/3/new.json 2024-04-12T17:04:02.746 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:04:02.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T17:04:02.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBaRlmKcgcKhAAPVdm0uXx/Hz2dy4qy/8/zg== --osd-uuid 1dcf5d98-e067-47c2-b383-8e29d196a9a6 2024-04-12T17:04:02.808 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:02.803+0000 7f77b3a337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:04:02.851 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:02.847+0000 7f77b3a337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:04:02.853 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:02.851+0000 7f77b3a337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:04:02.854 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:02.851+0000 7f77b3a337c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T17:04:02.854 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:02.851+0000 7f77b3a337c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T17:04:02.854 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:02.851+0000 7f77b3a337c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T17:04:03.010 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:03.007+0000 7f77b3a337c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T17:04:10.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T17:04:10.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:04:10.951 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T17:04:10.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T17:04:10.952 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:04:11.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T17:04:11.536 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T17:04:11.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:04:11.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:04:11.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:04:11.537 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:04:11.562 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:11.559+0000 7f84f069e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:04:11.564 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:11.563+0000 7f84f069e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:04:11.565 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:11.563+0000 7f84f069e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:04:11.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T17:04:11.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:04:11.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T17:04:11.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:04:11.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:04:11.949 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:11.949 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:04:11.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:04:11.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:11.950 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:12.110 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:12.107+0000 7f84f069e7c0 -1 Falling back to public interface 2024-04-12T17:04:12.406 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:13.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:13.408 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:13.408 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:04:13.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:04:13.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:13.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:13.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:14.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:14.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:14.833 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:04:14.834 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:04:14.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:14.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:15.237 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:04:15.235+0000 7f84f069e7c0 -1 osd.3 0 log_to_monitors true 2024-04-12T17:04:15.270 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:16.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:16.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:16.271 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:04:16.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:04:16.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:16.272 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:17.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:18.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:18.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:18.086 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:04:18.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:04:18.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:18.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:18.734 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:19.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:19.737 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:04:19.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:19.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:20.595 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:21.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:21.597 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:21.598 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:04:21.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:04:21.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:21.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:22.062 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:23.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:23.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:23.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:04:23.064 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:04:23.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:23.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:23.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:24.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:24.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:24.722 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:04:24.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:04:24.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:24.723 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:25.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:26.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:26.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:26.278 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:04:26.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:04:26.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:26.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:26.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:27.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:27.878 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:27.879 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:04:27.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:04:27.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:27.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:28.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:29.558 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:29.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:29.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:04:29.559 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:04:29.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:29.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:30.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:31.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:31.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:31.036 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:04:31.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:04:31.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:31.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:31.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:32.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:32.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:32.624 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:04:32.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:04:32.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:32.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:33.067 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:34.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:34.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:34.070 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:04:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:04:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:34.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:35.151 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:36.152 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:36.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:36.153 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:04:36.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:04:36.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:36.154 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:38.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:38.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:38.335 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:04:38.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:04:38.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:38.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:39.385 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:40.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:40.387 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:40.387 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:04:40.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:04:40.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:40.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:41.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:42.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:42.676 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:42.676 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:04:42.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:04:42.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:42.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:43.624 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:44.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:44.626 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:44.627 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:04:44.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:04:44.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:44.627 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:45.642 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:46.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:46.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:46.644 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:04:46.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:04:46.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:46.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:47.192 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:04:48.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:04:48.194 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:04:48.194 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:04:48.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:04:48.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:04:48.195 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:04:48.659 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2573377427,v1:127.0.0.1:6827/2573377427] [v2:127.0.0.1:6828/2573377427,v1:127.0.0.1:6829/2573377427] exists,up 1dcf5d98-e067-47c2-b383-8e29d196a9a6 2024-04-12T17:04:48.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:04:48.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:04:48.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:04:48.660 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T17:04:48.660 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T17:04:48.660 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T17:04:48.661 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:04:48.661 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:04:48.661 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:04:48.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T17:04:48.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T17:04:48.662 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T17:04:48.758 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T17:04:48.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T17:04:48.774 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T17:01:54.419+0000 7f56c2c707c0 0 load: jerasure load: lrc 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T17:04:48.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-04-12T17:04:49.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T17:04:49.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T17:04:51.640 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T17:04:51.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T17:04:52.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T17:04:52.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:04:52.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:04:52.661 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:04:52.661 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:04:52.662 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:04:52.663 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:04:52.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:04:52.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:04:52.663 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:04:52.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:04:52.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:04:52.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:04:52.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:04:52.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:04:52.790 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:04:53.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:04:53.212 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:04:53.212 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:04:53.213 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:04:53.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:04:53.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:04:53.213 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:04:53.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803813 2024-04-12T17:04:53.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803813 2024-04-12T17:04:53.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813' 2024-04-12T17:04:53.436 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:04:53.436 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:04:53.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574878 2024-04-12T17:04:53.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574878 2024-04-12T17:04:53.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574878' 2024-04-12T17:04:53.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:04:53.588 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:04:53.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378640 2024-04-12T17:04:53.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378640 2024-04-12T17:04:53.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574878 2-81604378640' 2024-04-12T17:04:53.788 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:04:53.788 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:04:53.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084296 2024-04-12T17:04:53.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084296 2024-04-12T17:04:53.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574878 2-81604378640 3-120259084296' 2024-04-12T17:04:53.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:04:53.935 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803813 2024-04-12T17:04:53.935 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:04:53.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:04:53.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803813 2024-04-12T17:04:53.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:04:53.937 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803813 2024-04-12T17:04:53.937 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803813 2024-04-12T17:04:53.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-04-12T17:04:53.938 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:04:54.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803813 -lt 25769803813 2024-04-12T17:04:54.374 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:04:54.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574878 2024-04-12T17:04:54.375 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:04:54.376 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:04:54.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574878 2024-04-12T17:04:54.377 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:04:54.378 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574878 2024-04-12T17:04:54.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574878' 2024-04-12T17:04:54.379 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574878 2024-04-12T17:04:54.379 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:04:54.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574878 -lt 55834574878 2024-04-12T17:04:54.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:04:54.832 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378640 2024-04-12T17:04:54.833 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:04:54.835 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:04:54.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378640 2024-04-12T17:04:54.836 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:04:54.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378640 2024-04-12T17:04:54.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378640' 2024-04-12T17:04:54.837 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378640 2024-04-12T17:04:54.837 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:04:55.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378638 -lt 81604378640 2024-04-12T17:04:55.257 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:04:56.259 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:04:56.259 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:04:56.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378640 -lt 81604378640 2024-04-12T17:04:56.672 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:04:56.673 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084296 2024-04-12T17:04:56.673 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:04:56.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:04:56.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084296 2024-04-12T17:04:56.676 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:04:56.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084296 2024-04-12T17:04:56.677 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 120259084296 2024-04-12T17:04:56.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084296' 2024-04-12T17:04:56.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:04:57.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084296 -lt 120259084296 2024-04-12T17:04:57.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:04:57.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:04:57.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:04:57.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:04:57.703 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:04:57.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:04:57.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:04:57.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:04:57.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:04:57.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:04:57.704 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:04:58.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:04:58.117 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:04:58.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:04:58.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:04:58.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:04:58.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:04:58.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79474-0-10 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79474-0-10 2024-04-12T17:04:58.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79474-0-10 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T17:04:58.684 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79474-0-10 td/test-erasure-eio/ORIGINAL 2024-04-12T17:04:58.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79474-0-10 td/test-erasure-eio 0 10 set 2024-04-12T17:04:58.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79474-0-10 2024-04-12T17:04:58.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-04-12T17:04:58.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-04-12T17:04:58.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79474-0-10 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79474-0-10 2024-04-12T17:04:58.884 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79474-0-10 2024-04-12T17:04:58.885 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:04:59.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:04:59.285 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:04:59.285 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T17:04:59.285 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-04-12T17:04:59.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-04-12T17:04:59.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-04-12T17:04:59.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-04-12T17:04:59.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-04-12T17:05:00.124 INFO:tasks.workunit.client.0.smithi137.stderr:noout is set 2024-04-12T17:05:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-04-12T17:05:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-04-12T17:05:00.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-04-12T17:05:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-04-12T17:05:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-04-12T17:05:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:10 bytes copied, 0.000127486 s, 78.4 kB/s 2024-04-12T17:05:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79474-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:05:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:05:00.146 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79474-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:05:00.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:05:00.148 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:05:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:05:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79474-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:05:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:05:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-12T17:05:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:05:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-04-12T17:05:01.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79474-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:05:03.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:05:03.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:05:03.738 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:05:03.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:05:03.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T17:05:03.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-12T17:05:03.740 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.3 2024-04-12T17:05:03.740 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:05:03.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-04-12T17:05:03.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-12T17:05:03.741 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:05:03.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:05:03.742 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:05:03.779 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:03.775+0000 7fc5897517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:03.804 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:03.803+0000 7fc5897517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:03.805 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:03.803+0000 7fc5897517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:04.170 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:05:04.366 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:04.363+0000 7fc5897517c0 -1 Falling back to public interface 2024-04-12T17:05:04.638 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:05.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:05.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:05.640 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:05:05.641 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:05.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:05.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:05:06.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:07.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:07.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:07.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:05:07.109 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:05:07.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:07.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:05:07.300 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:07.295+0000 7fc5897517c0 -1 osd.3 36 log_to_monitors true 2024-04-12T17:05:07.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:08.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:08.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:08.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:05:08.609 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:05:08.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:08.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:05:09.391 INFO:tasks.workunit.client.0.smithi137.stderr:osd.3 up in weight 1 up_from 42 up_thru 42 down_at 39 last_clean_interval [28,36) [v2:127.0.0.1:6826/1512133237,v1:127.0.0.1:6827/1512133237] [v2:127.0.0.1:6828/1512133237,v1:127.0.0.1:6829/1512133237] exists,up 1dcf5d98-e067-47c2-b383-8e29d196a9a6 2024-04-12T17:05:09.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:05:09.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:05:09.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:05:09.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:05:09.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:05:09.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:05:09.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:05:09.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:05:09.392 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:05:09.393 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:05:09.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:05:09.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:05:09.393 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:05:09.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:05:09.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:05:09.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:05:09.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:05:09.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:05:09.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:05:10.082 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:05:10.082 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:10.083 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:05:10.083 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:05:10.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:05:10.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:10.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:05:10.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803819 2024-04-12T17:05:10.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803819 2024-04-12T17:05:10.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803819' 2024-04-12T17:05:10.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:10.282 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:05:10.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574884 2024-04-12T17:05:10.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574884 2024-04-12T17:05:10.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803819 1-55834574884' 2024-04-12T17:05:10.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:10.435 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:05:10.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378645 2024-04-12T17:05:10.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378645 2024-04-12T17:05:10.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803819 1-55834574884 2-81604378645' 2024-04-12T17:05:10.639 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:10.639 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:05:10.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626435 2024-04-12T17:05:10.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626435 2024-04-12T17:05:10.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803819 1-55834574884 2-81604378645 3-180388626435' 2024-04-12T17:05:10.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:10.782 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803819 2024-04-12T17:05:10.782 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:10.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:05:10.783 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803819 2024-04-12T17:05:10.783 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:10.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803819 2024-04-12T17:05:10.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803819' 2024-04-12T17:05:10.784 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803819 2024-04-12T17:05:10.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:05:11.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803817 -lt 25769803819 2024-04-12T17:05:11.175 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:05:12.177 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:05:12.177 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:05:12.598 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803819 -lt 25769803819 2024-04-12T17:05:12.599 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:12.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574884 2024-04-12T17:05:12.600 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:12.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:05:12.602 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574884 2024-04-12T17:05:12.602 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:12.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574884 2024-04-12T17:05:12.603 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574884' 2024-04-12T17:05:12.603 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 55834574884 2024-04-12T17:05:12.603 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:05:13.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574884 -lt 55834574884 2024-04-12T17:05:13.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:13.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378645 2024-04-12T17:05:13.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:13.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:05:13.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378645 2024-04-12T17:05:13.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:13.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378645 2024-04-12T17:05:13.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378645' 2024-04-12T17:05:13.039 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378645 2024-04-12T17:05:13.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:05:13.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378646 -lt 81604378645 2024-04-12T17:05:13.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:13.428 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-180388626435 2024-04-12T17:05:13.428 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:13.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:05:13.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-180388626435 2024-04-12T17:05:13.430 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:13.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626435 2024-04-12T17:05:13.431 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 180388626435' 2024-04-12T17:05:13.432 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 180388626435 2024-04-12T17:05:13.432 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:05:13.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626435 -lt 180388626435 2024-04-12T17:05:13.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:05:13.885 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:05:13.886 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:05:14.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:05:14.430 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:05:14.431 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:05:14.431 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:05:14.431 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:05:14.431 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:05:14.431 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:05:14.431 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:05:14.821 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T17:05:14.821 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:05:14.822 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:05:14.822 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:05:15.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-12T17:05:15.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-12T17:05:15.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-12T17:05:15.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-12T17:05:15.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-12T17:05:15.336 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-12T17:05:15.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-12T17:05:15.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:05:15.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:05:15.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:05:15.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:05:15.438 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:05:15.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:05:15.439 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:05:15.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:05:15.827 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:05:15.827 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:05:15.827 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:05:16.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:05:16.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:05:16.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:05:16.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-04-12T17:05:16.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-04-12T17:05:17.407 INFO:tasks.workunit.client.0.smithi137.stderr:noout is unset 2024-04-12T17:05:17.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79474-0-10 2024-04-12T17:05:17.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:05:17.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:05:17.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79474-0-10 2024-04-12T17:05:17.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T17:05:17.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T17:05:17.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79474-0-10 td/test-erasure-eio/COPY 2024-04-12T17:05:17.797 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T17:05:17.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T17:05:17.802 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79474-0-10 td/test-erasure-eio 1 10 set 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79474-0-10 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-04-12T17:05:17.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-04-12T17:05:17.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-04-12T17:05:17.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-04-12T17:05:17.805 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79474-0-10 2024-04-12T17:05:17.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:05:17.806 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79474-0-10 2024-04-12T17:05:17.806 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79474-0-10 2024-04-12T17:05:17.806 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:05:18.252 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:05:18.252 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:18.252 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T17:05:18.252 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-04-12T17:05:18.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-04-12T17:05:18.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-04-12T17:05:18.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-04-12T17:05:18.252 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-04-12T17:05:19.146 INFO:tasks.workunit.client.0.smithi137.stderr:noout is set 2024-04-12T17:05:19.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-04-12T17:05:19.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-04-12T17:05:19.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-04-12T17:05:19.170 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-04-12T17:05:19.170 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-04-12T17:05:19.170 INFO:tasks.workunit.client.0.smithi137.stderr:10 bytes copied, 0.000199793 s, 50.1 kB/s 2024-04-12T17:05:19.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79474-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:19.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:05:19.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:05:19.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-12T17:05:19.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:05:19.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79474-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:19.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:05:19.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:05:19.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-12T17:05:19.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:05:19.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-04-12T17:05:19.172 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:05:19.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:05:19.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:05:19.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:05:19.173 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:05:20.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:05:20.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79474-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:20.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:05:20.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:05:20.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-12T17:05:20.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:05:20.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-04-12T17:05:20.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79474-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:23.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-04-12T17:05:23.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:05:23.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:05:23.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-12T17:05:23.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:05:23.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:05:23.005 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:05:23.007 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:05:23.007 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:05:23.007 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:05:23.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:05:23.007 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:05:23.008 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T17:05:23.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-12T17:05:23.009 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.1 2024-04-12T17:05:23.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:05:23.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-04-12T17:05:23.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-12T17:05:23.010 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:05:23.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:05:23.011 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:05:23.042 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:23.039+0000 7f8de95a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:23.045 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:23.043+0000 7f8de95a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:23.046 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:23.043+0000 7f8de95a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:23.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:05:23.603 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:23.599+0000 7f8de95a97c0 -1 Falling back to public interface 2024-04-12T17:05:23.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:24.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:24.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:24.866 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:05:24.866 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:24.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:24.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:05:25.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:26.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:26.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:26.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:05:26.280 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:05:26.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:26.281 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:05:26.632 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:26.627+0000 7f8de95a97c0 -1 osd.1 48 log_to_monitors true 2024-04-12T17:05:26.759 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:27.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:27.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:27.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:05:27.761 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:05:27.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:27.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:05:28.600 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 49 last_clean_interval [13,48) [v2:127.0.0.1:6810/3732670195,v1:127.0.0.1:6811/3732670195] [v2:127.0.0.1:6812/3732670195,v1:127.0.0.1:6813/3732670195] exists,up b0a91350-689e-430c-9299-6a4468635f7f 2024-04-12T17:05:28.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:05:28.600 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:05:28.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:05:28.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:05:28.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:05:28.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:05:28.601 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:05:28.602 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:05:28.602 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:05:28.602 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:05:28.602 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:05:28.602 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:05:28.602 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:05:28.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:05:28.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:05:28.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:05:28.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:05:28.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:05:28.722 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:05:29.262 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:05:29.262 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:29.262 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:05:29.262 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:05:29.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:05:29.265 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:29.266 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:05:29.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803825 2024-04-12T17:05:29.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803825 2024-04-12T17:05:29.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825' 2024-04-12T17:05:29.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:29.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:05:29.726 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299395 2024-04-12T17:05:29.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299395 2024-04-12T17:05:29.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-223338299395' 2024-04-12T17:05:29.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:29.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:05:29.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378651 2024-04-12T17:05:29.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378651 2024-04-12T17:05:29.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-223338299395 2-81604378651' 2024-04-12T17:05:29.876 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:29.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:05:30.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626441 2024-04-12T17:05:30.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626441 2024-04-12T17:05:30.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-223338299395 2-81604378651 3-180388626441' 2024-04-12T17:05:30.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:30.018 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803825 2024-04-12T17:05:30.018 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:30.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:05:30.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803825 2024-04-12T17:05:30.019 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:30.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803825 2024-04-12T17:05:30.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-04-12T17:05:30.021 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803825 2024-04-12T17:05:30.021 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:05:30.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803825 -lt 25769803825 2024-04-12T17:05:30.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:30.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299395 2024-04-12T17:05:30.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:30.445 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:05:30.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299395 2024-04-12T17:05:30.446 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299395 2024-04-12T17:05:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299395' 2024-04-12T17:05:30.447 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 223338299395 2024-04-12T17:05:30.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:05:30.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299395 -lt 223338299395 2024-04-12T17:05:30.859 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:30.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378651 2024-04-12T17:05:30.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:30.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:05:30.862 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378651 2024-04-12T17:05:30.863 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:30.863 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378651 2024-04-12T17:05:30.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378651' 2024-04-12T17:05:30.864 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378651 2024-04-12T17:05:30.864 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:05:31.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378649 -lt 81604378651 2024-04-12T17:05:31.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:05:32.318 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:05:32.319 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:05:32.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378652 -lt 81604378651 2024-04-12T17:05:32.724 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:32.724 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-180388626441 2024-04-12T17:05:32.725 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:32.727 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:05:32.727 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-180388626441 2024-04-12T17:05:32.728 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:32.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626441 2024-04-12T17:05:32.729 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 180388626441' 2024-04-12T17:05:32.729 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 180388626441 2024-04-12T17:05:32.729 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:05:33.123 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626441 -lt 180388626441 2024-04-12T17:05:33.123 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:05:33.124 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:05:33.124 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:05:33.709 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:05:33.710 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:05:33.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:05:33.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:05:33.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:05:33.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:05:33.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:05:33.711 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:05:34.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:05:34.158 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:05:34.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:05:34.159 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:05:34.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:05:34.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:05:34.754 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:05:34.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-04-12T17:05:34.755 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-04-12T17:05:35.519 INFO:tasks.workunit.client.0.smithi137.stderr:noout is unset 2024-04-12T17:05:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79474-0-10 fail 2024-04-12T17:05:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:05:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:05:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79474-0-10 2024-04-12T17:05:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-04-12T17:05:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-04-12T17:05:35.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79474-0-10 td/test-erasure-eio/COPY 2024-04-12T17:05:35.754 INFO:tasks.workunit.client.0.smithi137.stderr:error getting pool-jerasure/obj-size-79474-0-10: (5) Input/output error 2024-04-12T17:05:35.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-04-12T17:05:35.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-04-12T17:05:35.761 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79474-0-0 2024-04-12T17:05:35.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79474-0-0 2024-04-12T17:05:35.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T17:05:35.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T17:05:35.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79474-0-0 2024-04-12T17:05:35.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:05:35.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T17:05:35.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:05:35.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T17:05:35.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:05:35.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T17:05:35.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:05:35.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T17:05:35.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79474-0-0 td/test-erasure-eio/ORIGINAL 2024-04-12T17:05:36.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79474-0-0 td/test-erasure-eio 0 0 set 2024-04-12T17:05:36.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79474-0-0 2024-04-12T17:05:36.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-04-12T17:05:36.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-04-12T17:05:36.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79474-0-0 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79474-0-0 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79474-0-0 2024-04-12T17:05:36.037 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:05:36.505 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:05:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T17:05:36.506 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-04-12T17:05:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-04-12T17:05:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-04-12T17:05:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-04-12T17:05:36.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-04-12T17:05:37.309 INFO:tasks.workunit.client.0.smithi137.stderr:noout is set 2024-04-12T17:05:37.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-04-12T17:05:37.330 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-04-12T17:05:37.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-04-12T17:05:37.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79474-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:37.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:05:37.332 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:05:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-12T17:05:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:05:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79474-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:05:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:05:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-12T17:05:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:05:37.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-04-12T17:05:37.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:05:37.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:05:37.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:05:37.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:05:37.334 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:05:38.645 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:05:38.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79474-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:38.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:05:38.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:05:38.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-12T17:05:38.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:05:38.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-04-12T17:05:38.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79474-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:05:40.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:05:40.936 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:05:40.938 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:05:40.938 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:05:40.938 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:05:40.938 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:05:40.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:05:40.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:05:40.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:05:40.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-12T17:05:40.939 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.3 2024-04-12T17:05:40.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:05:40.940 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-04-12T17:05:40.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-12T17:05:40.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:05:40.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:05:40.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:05:40.973 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:40.971+0000 7f6b920707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:40.975 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:40.971+0000 7f6b920707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:40.977 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:40.975+0000 7f6b920707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:41.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-12T17:05:41.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:05:41.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T17:05:41.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:05:41.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:05:41.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:41.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:05:41.364 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:05:41.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:41.364 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:05:41.535 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:41.531+0000 7f6b920707c0 -1 Falling back to public interface 2024-04-12T17:05:41.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:42.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:42.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:42.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:05:42.777 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:42.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:42.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:05:43.183 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:05:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:05:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:05:44.507 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:44.503+0000 7f6b920707c0 -1 osd.3 58 log_to_monitors true 2024-04-12T17:05:44.615 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:05:45.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:05:45.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:45.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:05:45.618 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:05:45.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:45.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:05:46.212 INFO:tasks.workunit.client.0.smithi137.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [42,58) [v2:127.0.0.1:6826/3860355402,v1:127.0.0.1:6827/3860355402] [v2:127.0.0.1:6828/3860355402,v1:127.0.0.1:6829/3860355402] exists,up 1dcf5d98-e067-47c2-b383-8e29d196a9a6 2024-04-12T17:05:46.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:05:46.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:05:46.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:05:46.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:05:46.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:05:46.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:05:46.213 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:05:46.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:05:46.214 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:05:46.214 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:05:46.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:05:46.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:05:46.214 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:05:46.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:05:46.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:05:46.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:05:46.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:05:46.331 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:05:46.332 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:05:47.012 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:05:47.012 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:47.013 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:05:47.013 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:05:47.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:05:47.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:47.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803830 2024-04-12T17:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803830 2024-04-12T17:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830' 2024-04-12T17:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:47.340 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:05:47.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299401 2024-04-12T17:05:47.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299401 2024-04-12T17:05:47.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-223338299401' 2024-04-12T17:05:47.857 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:47.858 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:05:48.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378657 2024-04-12T17:05:48.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378657 2024-04-12T17:05:48.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-223338299401 2-81604378657' 2024-04-12T17:05:48.013 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:05:48.013 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:05:48.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972355 2024-04-12T17:05:48.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972355 2024-04-12T17:05:48.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-223338299401 2-81604378657 3-266287972355' 2024-04-12T17:05:48.165 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:48.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803830 2024-04-12T17:05:48.165 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:48.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:05:48.166 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803830 2024-04-12T17:05:48.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:48.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803830 2024-04-12T17:05:48.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-04-12T17:05:48.168 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803830 2024-04-12T17:05:48.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:05:48.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803830 -lt 25769803830 2024-04-12T17:05:48.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:48.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299401 2024-04-12T17:05:48.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:48.611 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:05:48.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299401 2024-04-12T17:05:48.612 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:48.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299401 2024-04-12T17:05:48.613 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299401' 2024-04-12T17:05:48.613 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 223338299401 2024-04-12T17:05:48.613 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:05:49.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299399 -lt 223338299401 2024-04-12T17:05:49.070 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:05:50.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:05:50.072 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:05:50.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299401 -lt 223338299401 2024-04-12T17:05:50.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:50.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378657 2024-04-12T17:05:50.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:50.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:05:50.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378657 2024-04-12T17:05:50.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:50.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378657 2024-04-12T17:05:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378657' 2024-04-12T17:05:50.528 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378657 2024-04-12T17:05:50.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:05:50.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378657 -lt 81604378657 2024-04-12T17:05:50.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:05:50.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972355 2024-04-12T17:05:50.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:05:50.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:05:50.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972355 2024-04-12T17:05:50.980 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:05:50.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972355 2024-04-12T17:05:50.982 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972355' 2024-04-12T17:05:50.982 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 266287972355 2024-04-12T17:05:50.982 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:05:51.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972355 -lt 266287972355 2024-04-12T17:05:51.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:05:51.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:05:51.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:05:51.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:05:51.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:05:51.992 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:05:51.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:05:51.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:05:51.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:05:51.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:05:51.993 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:05:52.405 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:05:52.405 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:05:52.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:05:52.406 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:05:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:05:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:05:52.960 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:05:52.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-04-12T17:05:52.961 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-04-12T17:05:53.710 INFO:tasks.workunit.client.0.smithi137.stderr:noout is unset 2024-04-12T17:05:53.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79474-0-0 2024-04-12T17:05:53.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:05:53.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:05:53.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79474-0-0 2024-04-12T17:05:53.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T17:05:53.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T17:05:53.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79474-0-0 td/test-erasure-eio/COPY 2024-04-12T17:05:53.967 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T17:05:53.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T17:05:53.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79474-0-0 td/test-erasure-eio 1 0 set 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79474-0-0 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-04-12T17:05:53.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-04-12T17:05:53.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-04-12T17:05:53.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-04-12T17:05:53.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-04-12T17:05:53.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79474-0-0 2024-04-12T17:05:53.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:05:53.991 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79474-0-0 2024-04-12T17:05:53.991 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79474-0-0 2024-04-12T17:05:53.992 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:05:54.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:05:54.413 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:05:54.413 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T17:05:54.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-04-12T17:05:54.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-04-12T17:05:54.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-04-12T17:05:54.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-04-12T17:05:54.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-04-12T17:05:55.165 INFO:tasks.workunit.client.0.smithi137.stderr:noout is set 2024-04-12T17:05:55.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-04-12T17:05:55.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-04-12T17:05:55.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-04-12T17:05:55.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79474-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:55.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:05:55.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:05:55.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79474-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:05:55.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:05:56.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:05:56.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79474-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:56.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:05:56.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:05:56.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-12T17:05:56.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:05:56.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-04-12T17:05:56.500 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79474-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:05:59.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-04-12T17:05:59.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T17:05:59.311 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:05:59.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:05:59.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:05:59.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:05:59.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:05:59.312 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:05:59.314 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:05:59.314 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:05:59.314 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:05:59.314 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:05:59.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:05:59.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-12T17:05:59.315 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.1 2024-04-12T17:05:59.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:05:59.316 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-04-12T17:05:59.316 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-12T17:05:59.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:05:59.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:05:59.317 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:05:59.347 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:59.343+0000 7fd1b055c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:59.350 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:59.347+0000 7fd1b055c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:59.351 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:59.347+0000 7fd1b055c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:05:59.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-12T17:05:59.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:05:59.748 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T17:05:59.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:05:59.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:05:59.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:05:59.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:05:59.749 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:05:59.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:05:59.749 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:05:59.906 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:05:59.903+0000 7fd1b055c7c0 -1 Falling back to public interface 2024-04-12T17:06:00.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:01.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:01.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:01.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:06:01.172 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:01.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:01.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:06:01.578 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:06:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:06:02.580 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:02.581 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:06:02.837 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:02.835+0000 7fd1b055c7c0 -1 osd.1 67 log_to_monitors true 2024-04-12T17:06:02.994 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:03.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:03.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:03.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:06:03.996 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:06:03.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:03.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:06:04.569 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [52,67) [v2:127.0.0.1:6810/2575784872,v1:127.0.0.1:6811/2575784872] [v2:127.0.0.1:6812/2575784872,v1:127.0.0.1:6813/2575784872] exists,up b0a91350-689e-430c-9299-6a4468635f7f 2024-04-12T17:06:04.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:06:04.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:06:04.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:06:04.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:06:04.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:06:04.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:06:04.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:06:04.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:06:04.570 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:06:04.570 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:06:04.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:06:04.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:06:04.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:06:04.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:06:04.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:06:04.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:06:04.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:06:04.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:06:04.677 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:06:05.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:06:05.577 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:05.577 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:06:05.577 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:06:05.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:06:05.577 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:05.578 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:06:05.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803836 2024-04-12T17:06:05.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803836 2024-04-12T17:06:05.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836' 2024-04-12T17:06:05.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:05.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:06:06.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612611 2024-04-12T17:06:06.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612611 2024-04-12T17:06:06.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-313532612611' 2024-04-12T17:06:06.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:06.185 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:06:06.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378663 2024-04-12T17:06:06.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378663 2024-04-12T17:06:06.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-313532612611 2-81604378663' 2024-04-12T17:06:06.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:06.381 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:06:06.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972361 2024-04-12T17:06:06.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972361 2024-04-12T17:06:06.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803836 1-313532612611 2-81604378663 3-266287972361' 2024-04-12T17:06:06.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:06.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803836 2024-04-12T17:06:06.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:06.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:06:06.536 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803836 2024-04-12T17:06:06.537 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803836 2024-04-12T17:06:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-04-12T17:06:06.538 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803836 2024-04-12T17:06:06.538 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:06:06.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803836 -lt 25769803836 2024-04-12T17:06:06.951 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:06.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-313532612611 2024-04-12T17:06:06.952 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:06.953 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:06:06.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:06.954 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-313532612611 2024-04-12T17:06:06.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612611 2024-04-12T17:06:06.955 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 313532612611' 2024-04-12T17:06:06.955 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 313532612611 2024-04-12T17:06:06.956 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:06:07.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 313532612611 2024-04-12T17:06:07.412 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:06:08.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:06:08.414 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:06:08.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612611 2024-04-12T17:06:08.844 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:08.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378663 2024-04-12T17:06:08.846 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:08.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:06:08.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378663 2024-04-12T17:06:08.848 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:08.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378663 2024-04-12T17:06:08.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378663' 2024-04-12T17:06:08.850 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378663 2024-04-12T17:06:08.850 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:06:09.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378663 -lt 81604378663 2024-04-12T17:06:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:09.299 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972361 2024-04-12T17:06:09.299 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:06:09.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972361 2024-04-12T17:06:09.301 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:09.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972361 2024-04-12T17:06:09.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972361' 2024-04-12T17:06:09.303 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 266287972361 2024-04-12T17:06:09.303 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:06:09.775 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972361 -lt 266287972361 2024-04-12T17:06:09.775 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:06:09.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:06:09.776 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:06:10.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:06:10.323 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:06:10.323 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:06:10.323 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:06:10.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:06:10.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:06:10.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:06:10.324 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:06:10.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:06:10.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:06:10.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:06:10.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:06:11.273 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:06:11.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:06:11.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:06:11.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-04-12T17:06:11.274 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-04-12T17:06:12.015 INFO:tasks.workunit.client.0.smithi137.stderr:noout is unset 2024-04-12T17:06:12.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79474-0-0 fail 2024-04-12T17:06:12.033 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:06:12.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:06:12.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79474-0-0 2024-04-12T17:06:12.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-04-12T17:06:12.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-04-12T17:06:12.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79474-0-0 td/test-erasure-eio/COPY 2024-04-12T17:06:12.158 INFO:tasks.workunit.client.0.smithi137.stderr:error getting pool-jerasure/obj-size-79474-0-0: (5) Input/output error 2024-04-12T17:06:12.162 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-04-12T17:06:12.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-04-12T17:06:12.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-04-12T17:06:12.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79474-0-256 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79474-0-256 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79474-0-256 2024-04-12T17:06:12.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:06:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T17:06:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:06:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T17:06:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:06:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T17:06:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:06:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T17:06:12.168 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79474-0-256 td/test-erasure-eio/ORIGINAL 2024-04-12T17:06:12.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79474-0-256 td/test-erasure-eio 0 256 add 2024-04-12T17:06:12.447 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79474-0-256 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79474-0-256 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:06:12.448 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79474-0-256 2024-04-12T17:06:12.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79474-0-256 2024-04-12T17:06:12.449 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:06:12.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:06:12.990 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:12.990 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T17:06:12.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-04-12T17:06:12.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-04-12T17:06:12.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-04-12T17:06:12.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-04-12T17:06:12.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-04-12T17:06:13.864 INFO:tasks.workunit.client.0.smithi137.stderr:noout is set 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79474-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79474-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-12T17:06:13.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:06:13.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-04-12T17:06:13.889 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:06:13.889 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:06:13.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:06:13.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:06:13.890 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:06:15.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:06:15.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79474-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:15.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:06:15.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:06:15.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-12T17:06:15.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:06:15.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-04-12T17:06:15.203 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79474-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:06:15.940 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T17:06:15.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T17:06:15.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:06:15.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:06:15.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:06:15.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:06:15.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:06:15.941 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:06:15.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:06:15.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T17:06:15.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-12T17:06:15.944 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.3 2024-04-12T17:06:15.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:06:15.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-04-12T17:06:15.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-12T17:06:15.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:06:15.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:06:15.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:06:15.982 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:15.979+0000 7fa5231da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:15.985 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:15.983+0000 7fa5231da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:15.986 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:15.983+0000 7fa5231da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:16.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-12T17:06:16.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:06:16.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T17:06:16.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:06:16.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:06:16.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:16.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:06:16.427 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:06:16.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:16.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:06:16.574 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:16.571+0000 7fa5231da7c0 -1 Falling back to public interface 2024-04-12T17:06:16.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:17.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:17.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:17.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:06:17.888 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:17.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:17.889 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:06:18.341 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:06:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:06:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:19.343 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:06:19.762 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:20.171 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:20.167+0000 7fa5231da7c0 -1 osd.3 77 log_to_monitors true 2024-04-12T17:06:20.763 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:20.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:20.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:06:20.764 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:06:20.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:20.764 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:06:21.568 INFO:tasks.workunit.client.0.smithi137.stderr:osd.3 up in weight 1 up_from 83 up_thru 73 down_at 80 last_clean_interval [62,77) [v2:127.0.0.1:6826/1046057196,v1:127.0.0.1:6827/1046057196] [v2:127.0.0.1:6828/1046057196,v1:127.0.0.1:6829/1046057196] exists,up 1dcf5d98-e067-47c2-b383-8e29d196a9a6 2024-04-12T17:06:21.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:06:21.568 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:06:21.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:06:21.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:06:21.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:06:21.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:06:21.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:06:21.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:06:21.570 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:06:21.570 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:06:21.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:06:21.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:06:21.570 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:06:21.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:06:21.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:06:21.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:06:21.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:06:21.691 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:06:21.691 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:06:22.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:06:22.335 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:22.335 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:06:22.335 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:06:22.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:06:22.335 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:22.336 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:06:22.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803841 2024-04-12T17:06:22.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803841 2024-04-12T17:06:22.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841' 2024-04-12T17:06:22.654 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:22.654 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:06:22.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612616 2024-04-12T17:06:22.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612616 2024-04-12T17:06:22.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-313532612616' 2024-04-12T17:06:22.807 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:22.807 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:06:22.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378668 2024-04-12T17:06:22.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378668 2024-04-12T17:06:22.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-313532612616 2-81604378668' 2024-04-12T17:06:22.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:22.964 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:06:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-04-12T17:06:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-04-12T17:06:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-313532612616 2-81604378668 3-356482285571' 2024-04-12T17:06:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:23.116 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803841 2024-04-12T17:06:23.116 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:23.117 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:06:23.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803841 2024-04-12T17:06:23.118 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:23.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803841 2024-04-12T17:06:23.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-04-12T17:06:23.119 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803841 2024-04-12T17:06:23.119 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:06:23.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803839 -lt 25769803841 2024-04-12T17:06:23.525 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:06:24.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:06:24.528 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:06:24.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803841 -lt 25769803841 2024-04-12T17:06:24.938 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:24.939 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-313532612616 2024-04-12T17:06:24.939 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:24.941 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:06:24.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-313532612616 2024-04-12T17:06:24.942 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:24.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612616 2024-04-12T17:06:24.942 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 313532612616' 2024-04-12T17:06:24.942 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 313532612616 2024-04-12T17:06:24.943 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:06:25.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612617 -lt 313532612616 2024-04-12T17:06:25.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:25.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378668 2024-04-12T17:06:25.410 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:25.411 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:06:25.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378668 2024-04-12T17:06:25.412 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:25.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378668 2024-04-12T17:06:25.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378668' 2024-04-12T17:06:25.413 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378668 2024-04-12T17:06:25.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:06:25.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378668 -lt 81604378668 2024-04-12T17:06:25.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:25.872 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-356482285571 2024-04-12T17:06:25.873 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:25.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:06:25.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-356482285571 2024-04-12T17:06:25.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:25.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-04-12T17:06:25.877 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 356482285571' 2024-04-12T17:06:25.877 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 356482285571 2024-04-12T17:06:25.877 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:06:26.333 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285571 2024-04-12T17:06:26.333 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:06:26.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:06:26.334 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:06:26.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:06:26.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:06:26.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:06:26.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:06:26.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:06:26.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:06:26.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:06:26.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:06:27.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:06:27.310 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:06:27.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:06:27.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:06:27.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:06:27.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:06:27.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:06:27.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:256 bytes copied, 7.0549e-05 s, 3.6 MB/s 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79474-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79474-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-12T17:06:27.925 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:06:27.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-04-12T17:06:27.926 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:06:27.926 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:06:27.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:06:27.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:06:27.926 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:06:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:06:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79474-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:06:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:06:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-12T17:06:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:06:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-04-12T17:06:28.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79474-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:31.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-04-12T17:06:31.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:06:31.544 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:06:31.545 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:06:31.546 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:06:31.546 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:06:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:06:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:06:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:06:31.547 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:06:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:06:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:06:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:06:31.547 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:06:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:06:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:06:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:06:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:06:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:06:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:06:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T17:06:31.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-12T17:06:31.549 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.3 2024-04-12T17:06:31.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:06:31.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-04-12T17:06:31.550 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-12T17:06:31.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:06:31.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:06:31.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:06:31.582 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:31.579+0000 7f19c312e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:31.585 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:31.583+0000 7f19c312e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:31.587 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:31.583+0000 7f19c312e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:31.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:06:31.981 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:32.133 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:32.131+0000 7f19c312e7c0 -1 Falling back to public interface 2024-04-12T17:06:32.458 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:33.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:33.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:33.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:06:33.460 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:33.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:33.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:06:33.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:34.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:34.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:34.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:06:34.887 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:06:34.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:34.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:06:34.941 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:34.939+0000 7f19c312e7c0 -1 osd.3 84 log_to_monitors true 2024-04-12T17:06:35.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:36.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:36.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:36.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:06:36.393 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:06:36.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:36.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:06:37.023 INFO:tasks.workunit.client.0.smithi137.stderr:osd.3 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [83,84) [v2:127.0.0.1:6826/2436921001,v1:127.0.0.1:6827/2436921001] [v2:127.0.0.1:6828/2436921001,v1:127.0.0.1:6829/2436921001] exists,up 1dcf5d98-e067-47c2-b383-8e29d196a9a6 2024-04-12T17:06:37.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:06:37.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:06:37.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:06:37.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:06:37.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:06:37.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:06:37.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:06:37.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:06:37.025 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:06:37.025 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:06:37.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:06:37.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:06:37.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:06:37.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:06:37.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:06:37.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:06:37.142 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:06:37.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:06:37.143 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:06:37.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:06:37.567 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:37.567 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:06:37.567 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:06:37.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:06:37.567 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:37.568 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:06:37.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803846 2024-04-12T17:06:37.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803846 2024-04-12T17:06:37.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803846' 2024-04-12T17:06:37.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:37.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:06:37.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612621 2024-04-12T17:06:37.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612621 2024-04-12T17:06:37.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803846 1-313532612621' 2024-04-12T17:06:37.928 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:37.928 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:06:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378673 2024-04-12T17:06:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378673 2024-04-12T17:06:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803846 1-313532612621 2-81604378673' 2024-04-12T17:06:38.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:38.084 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:06:38.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122051 2024-04-12T17:06:38.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122051 2024-04-12T17:06:38.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803846 1-313532612621 2-81604378673 3-377957122051' 2024-04-12T17:06:38.280 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:38.281 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803846 2024-04-12T17:06:38.281 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:38.282 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:06:38.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803846 2024-04-12T17:06:38.283 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803846 2024-04-12T17:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803846' 2024-04-12T17:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803846 2024-04-12T17:06:38.284 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:06:38.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803845 -lt 25769803846 2024-04-12T17:06:38.692 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:06:39.694 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:06:39.695 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:06:40.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803847 -lt 25769803846 2024-04-12T17:06:40.135 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:40.136 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-313532612621 2024-04-12T17:06:40.137 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:40.138 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:06:40.139 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-313532612621 2024-04-12T17:06:40.139 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:40.140 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612621 2024-04-12T17:06:40.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 313532612621' 2024-04-12T17:06:40.141 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 313532612621 2024-04-12T17:06:40.141 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:06:40.622 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612622 -lt 313532612621 2024-04-12T17:06:40.623 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:40.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378673 2024-04-12T17:06:40.624 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:40.625 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:06:40.626 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378673 2024-04-12T17:06:40.626 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:40.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378673 2024-04-12T17:06:40.628 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378673' 2024-04-12T17:06:40.628 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378673 2024-04-12T17:06:40.628 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:06:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378673 -lt 81604378673 2024-04-12T17:06:41.095 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:41.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:41.096 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122051 2024-04-12T17:06:41.097 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:06:41.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:41.098 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122051 2024-04-12T17:06:41.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122051 2024-04-12T17:06:41.100 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122051' 2024-04-12T17:06:41.100 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 377957122051 2024-04-12T17:06:41.100 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:06:41.557 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122051 2024-04-12T17:06:41.558 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:06:41.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:06:41.559 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:06:42.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:06:42.088 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:06:42.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:06:42.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:06:42.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:06:42.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:06:42.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:06:42.088 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:06:42.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:06:42.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:06:42.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:06:42.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:06:43.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:06:43.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:06:43.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:06:43.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-04-12T17:06:43.064 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-04-12T17:06:43.831 INFO:tasks.workunit.client.0.smithi137.stderr:noout is unset 2024-04-12T17:06:43.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79474-0-256 2024-04-12T17:06:43.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:06:43.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:06:43.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79474-0-256 2024-04-12T17:06:43.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T17:06:43.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T17:06:43.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79474-0-256 td/test-erasure-eio/COPY 2024-04-12T17:06:44.158 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T17:06:44.160 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T17:06:44.183 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-04-12T17:06:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-04-12T17:06:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79474-0-256 td/test-erasure-eio 1 256 add 2024-04-12T17:06:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79474-0-256 2024-04-12T17:06:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-04-12T17:06:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79474-0-256 2024-04-12T17:06:44.186 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:06:44.187 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79474-0-256 2024-04-12T17:06:44.187 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79474-0-256 2024-04-12T17:06:44.187 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:06:44.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:06:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T17:06:44.621 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-04-12T17:06:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-04-12T17:06:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-04-12T17:06:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-04-12T17:06:44.621 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-04-12T17:06:45.404 INFO:tasks.workunit.client.0.smithi137.stderr:noout is set 2024-04-12T17:06:45.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-04-12T17:06:45.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79474-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:45.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:06:45.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:06:45.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-12T17:06:45.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:06:45.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79474-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:45.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:06:45.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:06:45.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-12T17:06:45.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:06:45.423 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-04-12T17:06:45.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:06:45.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:06:45.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:06:45.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:06:45.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:06:46.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:06:46.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79474-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:46.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:06:46.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:06:46.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-12T17:06:46.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:06:46.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-04-12T17:06:46.737 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79474-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:47.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-04-12T17:06:47.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:06:47.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:06:47.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:06:47.390 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:06:47.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.1 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:06:47.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:06:47.426 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:47.423+0000 7fb9d15de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:47.428 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:47.427+0000 7fb9d15de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:47.430 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:47.427+0000 7fb9d15de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:06:47.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-12T17:06:47.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:06:47.823 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T17:06:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:06:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:06:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:06:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:06:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:47.824 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:06:47.977 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:47.975+0000 7fb9d15de7c0 -1 Falling back to public interface 2024-04-12T17:06:48.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:49.233 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:49.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:49.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:06:49.234 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:49.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:49.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:06:49.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:50.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:50.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:50.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:06:50.646 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:06:50.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:50.647 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:06:51.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:06:51.587 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:06:51.583+0000 7fb9d15de7c0 -1 osd.1 93 log_to_monitors true 2024-04-12T17:06:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:06:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:06:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:06:52.073 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:06:52.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:06:52.074 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:06:52.735 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 up in weight 1 up_from 97 up_thru 85 down_at 94 last_clean_interval [73,93) [v2:127.0.0.1:6810/3004707756,v1:127.0.0.1:6811/3004707756] [v2:127.0.0.1:6812/3004707756,v1:127.0.0.1:6813/3004707756] exists,up b0a91350-689e-430c-9299-6a4468635f7f 2024-04-12T17:06:52.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:06:52.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:06:52.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:06:52.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:06:52.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:06:52.735 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:06:52.736 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:06:52.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:06:52.736 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:06:52.736 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:06:52.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:06:52.736 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:06:52.737 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:06:52.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:06:52.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:06:52.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:06:52.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:06:52.853 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:06:52.853 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:06:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:06:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:06:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:06:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:06:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:06:53.379 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:53.380 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:06:53.643 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803851 2024-04-12T17:06:53.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803851 2024-04-12T17:06:53.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851' 2024-04-12T17:06:53.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:53.644 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:06:53.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-04-12T17:06:53.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-04-12T17:06:53.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-416611827715' 2024-04-12T17:06:53.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:53.872 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:06:54.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378678 2024-04-12T17:06:54.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378678 2024-04-12T17:06:54.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-416611827715 2-81604378678' 2024-04-12T17:06:54.271 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:06:54.271 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:06:54.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122056 2024-04-12T17:06:54.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122056 2024-04-12T17:06:54.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803851 1-416611827715 2-81604378678 3-377957122056' 2024-04-12T17:06:54.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:54.494 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803851 2024-04-12T17:06:54.494 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:54.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:06:54.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803851 2024-04-12T17:06:54.496 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:54.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803851 2024-04-12T17:06:54.497 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803851' 2024-04-12T17:06:54.497 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803851 2024-04-12T17:06:54.498 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:06:55.149 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803851 -lt 25769803851 2024-04-12T17:06:55.150 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:55.150 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-416611827715 2024-04-12T17:06:55.151 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:55.153 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:06:55.154 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-416611827715 2024-04-12T17:06:55.155 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:55.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-04-12T17:06:55.157 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 416611827715' 2024-04-12T17:06:55.157 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 416611827715 2024-04-12T17:06:55.157 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:06:55.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 416611827715 2024-04-12T17:06:55.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:06:56.608 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:06:56.609 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:06:57.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827715 2024-04-12T17:06:57.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:57.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378678 2024-04-12T17:06:57.035 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:57.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:06:57.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378678 2024-04-12T17:06:57.038 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:57.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378678 2024-04-12T17:06:57.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378678' 2024-04-12T17:06:57.040 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378678 2024-04-12T17:06:57.040 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:06:57.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378678 -lt 81604378678 2024-04-12T17:06:57.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:06:57.518 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122056 2024-04-12T17:06:57.518 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:06:57.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:06:57.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122056 2024-04-12T17:06:57.522 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:06:57.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122056 2024-04-12T17:06:57.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122056' 2024-04-12T17:06:57.523 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 377957122056 2024-04-12T17:06:57.523 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:06:57.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122057 -lt 377957122056 2024-04-12T17:06:57.977 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:06:57.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:06:57.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:06:58.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:06:58.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:06:58.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:06:58.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:06:58.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:06:58.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:06:58.525 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:06:58.526 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:06:58.913 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:06:58.913 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:06:58.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:06:58.914 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:06:59.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:06:59.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:06:59.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:06:59.441 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-04-12T17:06:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-04-12T17:06:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-04-12T17:06:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:256 bytes copied, 4.6829e-05 s, 5.5 MB/s 2024-04-12T17:06:59.442 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79474-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79474-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:06:59.443 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-04-12T17:06:59.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:06:59.444 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:06:59.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:06:59.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:06:59.444 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:06:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:06:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79474-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:06:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:06:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:06:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-12T17:06:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:06:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-04-12T17:06:59.753 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79474-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:07:03.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-04-12T17:07:03.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:07:03.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:07:03.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-12T17:07:03.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:07:03.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T17:07:03.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:07:03.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:07:03.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:07:03.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:07:03.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T17:07:03.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T17:07:03.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:07:03.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:07:03.529 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:07:03.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:07:03.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:07:03.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:07:03.530 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:07:03.530 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:03.530 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:03.530 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:07:03.531 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:07:03.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:07:03.533 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:07:03.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:07:03.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:07:03.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:07:03.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:07:03.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:07:03.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:07:03.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:07:03.534 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T17:07:03.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-12T17:07:03.535 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.1 2024-04-12T17:07:03.535 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:07:03.535 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-04-12T17:07:03.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-12T17:07:03.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:07:03.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:07:03.536 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:07:03.567 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:03.563+0000 7ff9873737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:03.570 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:03.567+0000 7ff9873737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:03.572 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:03.571+0000 7ff9873737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:03.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-12T17:07:03.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:07:03.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T17:07:03.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:07:03.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:07:03.976 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:03.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:07:03.977 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:07:03.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:03.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:07:04.146 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:04.143+0000 7ff9873737c0 -1 Falling back to public interface 2024-04-12T17:07:04.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:07:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:07:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:05.396 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:07:05.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:06.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:06.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:06.813 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:07:06.813 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:07:06.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:06.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:07:06.996 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:06.991+0000 7ff9873737c0 -1 osd.1 99 log_to_monitors true 2024-04-12T17:07:07.297 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:08.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:08.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:08.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:07:08.299 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:07:08.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:08.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:07:09.042 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 up in weight 1 up_from 104 up_thru 104 down_at 100 last_clean_interval [97,99) [v2:127.0.0.1:6810/1416626945,v1:127.0.0.1:6811/1416626945] [v2:127.0.0.1:6812/1416626945,v1:127.0.0.1:6813/1416626945] exists,up b0a91350-689e-430c-9299-6a4468635f7f 2024-04-12T17:07:09.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:07:09.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:07:09.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:07:09.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:07:09.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:07:09.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:07:09.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:07:09.043 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:07:09.044 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:07:09.044 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:07:09.044 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:07:09.044 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:07:09.044 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:07:09.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:07:09.163 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:07:09.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:07:09.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:07:09.164 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:07:09.164 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:07:09.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:07:09.659 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:07:09.659 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:07:09.659 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:07:09.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:07:09.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:07:09.660 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:07:09.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803857 2024-04-12T17:07:09.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803857 2024-04-12T17:07:09.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857' 2024-04-12T17:07:09.811 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:07:09.811 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:07:09.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-04-12T17:07:09.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-04-12T17:07:09.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-446676598787' 2024-04-12T17:07:09.968 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:07:09.969 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:07:10.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378683 2024-04-12T17:07:10.124 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378683 2024-04-12T17:07:10.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-446676598787 2-81604378683' 2024-04-12T17:07:10.125 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:07:10.125 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:07:10.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122062 2024-04-12T17:07:10.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122062 2024-04-12T17:07:10.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-446676598787 2-81604378683 3-377957122062' 2024-04-12T17:07:10.310 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:07:10.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803857 2024-04-12T17:07:10.311 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:07:10.312 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:07:10.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803857 2024-04-12T17:07:10.313 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:07:10.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803857 2024-04-12T17:07:10.314 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803857' 2024-04-12T17:07:10.314 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803857 2024-04-12T17:07:10.315 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:07:10.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803855 -lt 25769803857 2024-04-12T17:07:10.745 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:07:11.747 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:07:11.747 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:07:12.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803857 -lt 25769803857 2024-04-12T17:07:12.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:07:12.169 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-446676598787 2024-04-12T17:07:12.170 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:07:12.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:07:12.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-446676598787 2024-04-12T17:07:12.173 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:07:12.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-04-12T17:07:12.174 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 446676598787' 2024-04-12T17:07:12.174 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 446676598787 2024-04-12T17:07:12.174 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:07:12.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598787 2024-04-12T17:07:12.641 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:07:12.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378683 2024-04-12T17:07:12.642 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:07:12.644 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:07:12.645 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378683 2024-04-12T17:07:12.645 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:07:12.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378683 2024-04-12T17:07:12.646 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378683' 2024-04-12T17:07:12.646 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378683 2024-04-12T17:07:12.647 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:07:13.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378684 -lt 81604378683 2024-04-12T17:07:13.093 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:07:13.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122062 2024-04-12T17:07:13.094 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:07:13.096 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:07:13.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122062 2024-04-12T17:07:13.097 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:07:13.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122062 2024-04-12T17:07:13.098 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122062' 2024-04-12T17:07:13.098 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 377957122062 2024-04-12T17:07:13.099 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:07:13.542 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122062 -lt 377957122062 2024-04-12T17:07:13.546 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:07:13.546 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:07:13.546 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:07:14.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:07:14.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:07:14.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:07:14.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:07:14.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:07:14.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:07:14.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:07:14.103 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:07:14.508 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T17:07:14.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:07:14.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:07:14.509 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:07:15.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-12T17:07:15.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-12T17:07:15.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-12T17:07:15.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-12T17:07:15.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-12T17:07:15.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-12T17:07:15.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-12T17:07:15.147 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:07:15.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:07:15.148 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:07:15.149 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:07:15.149 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:07:15.149 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:07:15.149 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:07:15.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-12T17:07:15.561 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:07:15.561 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:07:15.561 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-12T17:07:16.077 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-12T17:07:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-12T17:07:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-12T17:07:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-12T17:07:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-12T17:07:16.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-12T17:07:16.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-12T17:07:16.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:07:16.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:07:16.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:07:16.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:07:16.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:07:16.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:07:16.876 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:07:17.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:07:17.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:07:17.275 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:07:17.276 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:07:17.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:07:17.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:07:17.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:07:17.801 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-04-12T17:07:17.802 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-04-12T17:07:18.551 INFO:tasks.workunit.client.0.smithi137.stderr:noout is unset 2024-04-12T17:07:18.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79474-0-256 fail 2024-04-12T17:07:18.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:07:18.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:07:18.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79474-0-256 2024-04-12T17:07:18.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-04-12T17:07:18.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-04-12T17:07:18.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79474-0-256 td/test-erasure-eio/COPY 2024-04-12T17:07:18.827 INFO:tasks.workunit.client.0.smithi137.stderr:error getting pool-jerasure/obj-size-79474-0-256: (5) Input/output error 2024-04-12T17:07:18.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-04-12T17:07:18.832 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-04-12T17:07:18.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:312: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-04-12T17:07:18.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T17:07:18.834 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-04-12T17:07:19.513 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' does not exist 2024-04-12T17:07:19.530 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-04-12T17:07:20.663 INFO:tasks.workunit.client.0.smithi137.stderr:erasure-code-profile myprofile does not exist 2024-04-12T17:07:20.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-04-12T17:07:20.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T17:07:20.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T17:07:20.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T17:07:20.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:07:20.681 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:07:20.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:07:20.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:07:20.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:07:20.990 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:07:20.990 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T17:07:20.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T17:07:20.992 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T17:07:20.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T17:07:20.993 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T17:07:20.994 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T17:07:20.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:07:20.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T17:07:20.995 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T17:07:20.996 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:07:20.996 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T17:07:20.998 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T17:07:20.999 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T17:07:21.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T17:07:21.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:21.034 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:21.034 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T17:07:21.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T17:07:21.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T17:07:21.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-04-12T17:07:21.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-04-12T17:07:21.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-04-12T17:07:21.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-04-12T17:07:21.036 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T17:07:21.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-12T17:07:21.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T17:07:21.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:07:21.037 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:07:21.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:07:21.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:07:21.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:07:21.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:07:21.039 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T17:07:21.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T17:07:21.040 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T17:07:21.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T17:07:21.042 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T17:07:21.042 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T17:07:21.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:07:21.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T17:07:21.043 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T17:07:21.044 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:07:21.045 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T17:07:21.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-12T17:07:21.046 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T17:07:21.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T17:07:21.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:21.047 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:21.047 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T17:07:21.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T17:07:21.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T17:07:21.048 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-04-12T17:07:21.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-12T17:07:21.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:21.050 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:21.050 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79474 2024-04-12T17:07:21.052 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-12T17:07:21.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-12T17:07:21.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-12T17:07:21.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-04-12T17:07:21.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-04-12T17:07:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-04-12T17:07:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-12T17:07:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-12T17:07:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-12T17:07:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-04-12T17:07:21.053 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-04-12T17:07:21.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-12T17:07:21.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:07:21.511 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:07:21.511 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:07:21.512 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:21.512 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:21.512 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:07:21.512 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-12T17:07:21.739 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-12T17:07:21.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-12T17:07:21.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T17:07:21.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T17:07:21.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-12T17:07:21.740 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-12T17:07:21.741 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T17:07:21.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T17:07:21.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T17:07:21.742 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:07:21.742 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:21.742 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:21.742 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T17:07:21.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T17:07:21.742 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get fsid 2024-04-12T17:07:21.858 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-12T17:07:21.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-12T17:07:21.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-12T17:07:21.859 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-12T17:07:21.860 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-12T17:07:21.860 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-12T17:07:21.860 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T17:07:21.860 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T17:07:21.860 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:07:21.860 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:21.861 INFO:tasks.workunit.client.0.smithi137.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:21.861 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T17:07:21.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-12T17:07:21.861 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79474/ceph-mon.a.asok config get mon_host 2024-04-12T17:07:21.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-04-12T17:07:21.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-04-12T17:07:21.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-12T17:07:21.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-12T17:07:21.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-12T17:07:21.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-04-12T17:07:21.974 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-12T17:07:22.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-12T17:07:22.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:07:22.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:07:22.944 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:07:22.944 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:22.944 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:22.944 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:07:22.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T17:07:22.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-12T17:07:22.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-04-12T17:07:22.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-12T17:07:23.350 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'rbd' already exists 2024-04-12T17:07:23.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T17:07:24.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-04-12T17:07:24.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-12T17:07:24.371 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-12T17:07:24.372 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:07:24.372 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:24.372 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:24.372 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-mon.a.asok 2024-04-12T17:07:24.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-04-12T17:07:24.373 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-mon.a.asok log flush 2024-04-12T17:07:24.474 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T17:07:24.487 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-04-12T17:07:24.489 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T17:07:21.551+0000 7f1284ebdc80 0 load: jerasure load: lrc 2024-04-12T17:07:24.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-04-12T17:07:24.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-04-12T17:07:24.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:317: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-04-12T17:07:24.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-04-12T17:07:24.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-04-12T17:07:24.491 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-04-12T17:07:24.493 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:07:24.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:07:24.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:07:24.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:07:24.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-04-12T17:07:24.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-04-12T17:07:24.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:07:24.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:07:24.495 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:07:24.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:07:24.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:07:24.497 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:07:24.497 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:07:24.497 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:07:24.498 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:07:24.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-04-12T17:07:24.499 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:07:24.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46321fc1-760c-46f0-82d7-2fc1340dd093 2024-04-12T17:07:24.499 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 46321fc1-760c-46f0-82d7-2fc1340dd093' 2024-04-12T17:07:24.499 INFO:tasks.workunit.client.0.smithi137.stdout:add osd0 46321fc1-760c-46f0-82d7-2fc1340dd093 2024-04-12T17:07:24.500 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:07:24.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMahlmoXqpHhAABE3ghWi5s2ClpvgBUWyTXw== 2024-04-12T17:07:24.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMahlmoXqpHhAABE3ghWi5s2ClpvgBUWyTXw=="}' 2024-04-12T17:07:24.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46321fc1-760c-46f0-82d7-2fc1340dd093 -i td/test-erasure-eio/0/new.json 2024-04-12T17:07:24.868 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:07:24.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-04-12T17:07:24.886 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMahlmoXqpHhAABE3ghWi5s2ClpvgBUWyTXw== --osd-uuid 46321fc1-760c-46f0-82d7-2fc1340dd093 2024-04-12T17:07:24.920 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:24.919+0000 7ff6f2fd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:24.954 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:24.951+0000 7ff6f2fd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:24.955 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:24.951+0000 7ff6f2fd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:24.956 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:24.951+0000 7ff6f2fd37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T17:07:24.956 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:24.951+0000 7ff6f2fd37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T17:07:24.956 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:24.951+0000 7ff6f2fd37c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-04-12T17:07:24.993 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:24.991+0000 7ff6f2fd37c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-04-12T17:07:33.114 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-04-12T17:07:33.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:07:33.116 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd0 key to auth repository 2024-04-12T17:07:33.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-12T17:07:33.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:07:33.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-12T17:07:33.686 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.0 2024-04-12T17:07:33.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:07:33.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:07:33.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:07:33.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:07:33.717 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:33.715+0000 7f1c354ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:33.719 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:33.715+0000 7f1c354ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:33.721 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:33.719+0000 7f1c354ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:07:34.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-12T17:07:34.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:07:34.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-12T17:07:34.102 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:07:34.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:07:34.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:34.103 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:07:34.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:07:34.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:34.103 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:34.265 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:34.263+0000 7f1c354ed7c0 -1 Falling back to public interface 2024-04-12T17:07:34.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:35.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:35.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:35.521 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:07:35.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:07:35.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:35.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:35.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:36.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:36.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:36.946 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:07:36.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:07:36.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:36.946 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:37.124 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:07:37.119+0000 7f1c354ed7c0 -1 osd.0 0 log_to_monitors true 2024-04-12T17:07:37.391 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:38.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:38.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:38.393 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:07:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:07:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:38.394 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:38.836 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:39.837 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:39.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:39.838 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:07:39.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:07:39.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:39.838 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:40.349 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:41.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:41.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:41.352 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:07:41.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:07:41.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:41.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:41.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:42.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:42.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:42.785 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:07:42.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:07:42.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:42.786 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:43.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:44.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:44.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:44.302 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:07:44.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:07:44.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:44.303 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:44.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:45.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:45.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:45.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:07:45.789 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:07:45.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:45.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:46.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:47.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:47.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:47.352 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:07:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:07:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:47.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:47.846 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:48.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:48.848 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:48.849 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:07:48.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:07:48.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:48.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:49.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:50.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:50.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:50.304 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:07:50.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:07:50.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:50.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:50.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:51.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:51.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:51.888 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:07:51.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:07:51.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:51.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:52.381 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:53.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:53.383 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:53.383 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:07:53.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:07:53.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:53.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:53.858 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:54.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:54.860 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:54.860 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:07:54.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:07:54.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:54.861 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:55.291 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:56.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:56.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:56.293 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:07:56.293 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:07:56.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:56.294 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:57.719 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:07:58.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:07:58.721 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:07:58.722 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:07:58.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:07:58.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:07:58.722 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:07:59.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:00.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:00.682 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:00.682 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:08:00.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:08:00.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:00.683 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:08:01.426 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:02.427 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:02.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:02.428 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:08:02.428 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:08:02.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:02.429 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:08:03.283 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:04.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:04.285 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:04.285 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:08:04.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:08:04.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:04.286 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:08:05.787 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:06.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:06.789 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:06.789 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:08:06.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:08:06.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:06.790 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:08:07.800 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:08.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:08.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:08.804 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:08:08.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:08:08.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:08.804 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-12T17:08:09.295 INFO:tasks.workunit.client.0.smithi137.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3952477689,v1:127.0.0.1:6803/3952477689] [v2:127.0.0.1:6804/3952477689,v1:127.0.0.1:6805/3952477689] exists,up 46321fc1-760c-46f0-82d7-2fc1340dd093 2024-04-12T17:08:09.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:08:09.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T17:08:09.299 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:08:09.300 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:08:09.301 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T17:08:09.302 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:08:09.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=013a7c53-97c7-481e-aa2c-1d43a0d425a7 2024-04-12T17:08:09.302 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 013a7c53-97c7-481e-aa2c-1d43a0d425a7' 2024-04-12T17:08:09.302 INFO:tasks.workunit.client.0.smithi137.stdout:add osd1 013a7c53-97c7-481e-aa2c-1d43a0d425a7 2024-04-12T17:08:09.302 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:08:09.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5ahlmNFz7EhAAxzpkyl44Jc8g5EWB5rqhiQ== 2024-04-12T17:08:09.320 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5ahlmNFz7EhAAxzpkyl44Jc8g5EWB5rqhiQ=="}' 2024-04-12T17:08:09.321 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 013a7c53-97c7-481e-aa2c-1d43a0d425a7 -i td/test-erasure-eio/1/new.json 2024-04-12T17:08:10.466 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:08:10.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-04-12T17:08:10.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5ahlmNFz7EhAAxzpkyl44Jc8g5EWB5rqhiQ== --osd-uuid 013a7c53-97c7-481e-aa2c-1d43a0d425a7 2024-04-12T17:08:10.526 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:10.523+0000 7f697e7ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:10.548 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:10.543+0000 7f697e7ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:10.549 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:10.547+0000 7f697e7ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:10.549 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:10.547+0000 7f697e7ff7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T17:08:10.550 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:10.547+0000 7f697e7ff7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T17:08:10.550 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:10.547+0000 7f697e7ff7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-04-12T17:08:10.637 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:10.635+0000 7f697e7ff7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-04-12T17:08:18.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-04-12T17:08:18.934 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:08:18.935 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd1 key to auth repository 2024-04-12T17:08:18.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-12T17:08:18.935 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:08:19.516 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-12T17:08:19.516 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.1 2024-04-12T17:08:19.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:08:19.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:08:19.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:08:19.517 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:08:19.547 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:19.543+0000 7fc3044877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:19.550 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:19.547+0000 7fc3044877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:19.551 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:19.547+0000 7fc3044877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:19.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-12T17:08:19.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:08:19.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T17:08:19.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:08:19.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:08:19.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:19.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:08:19.916 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:08:19.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:19.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:20.098 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:20.095+0000 7fc3044877c0 -1 Falling back to public interface 2024-04-12T17:08:20.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:21.351 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:21.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:21.352 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:08:21.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:08:21.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:21.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:21.776 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:22.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:22.778 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:22.778 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:08:22.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:08:22.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:22.779 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:23.033 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:23.031+0000 7fc3044877c0 -1 osd.1 0 log_to_monitors true 2024-04-12T17:08:23.206 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:24.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:24.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:24.208 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:08:24.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:08:24.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:24.209 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:24.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:25.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:25.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:25.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:08:25.773 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:08:25.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:25.774 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:26.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:27.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:27.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:27.473 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:08:27.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:08:27.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:27.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:28.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:29.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:29.171 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:29.171 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:08:29.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:08:29.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:29.172 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:29.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:30.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:30.618 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:30.618 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:08:30.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:08:30.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:30.619 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:31.322 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:32.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:32.324 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:32.324 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:08:32.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:08:32.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:32.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:32.803 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:33.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:33.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:33.805 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:08:33.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:08:33.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:33.806 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:34.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:35.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:35.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:35.371 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:08:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:08:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:35.372 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:35.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:36.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:36.887 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:36.887 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:08:36.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:08:36.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:36.888 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:37.473 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:38.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:38.474 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:38.475 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:08:38.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:08:38.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:38.475 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:39.212 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:40.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:40.214 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:40.214 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:08:40.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:08:40.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:40.215 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:40.685 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:41.686 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:41.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:41.687 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:08:41.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:08:41.687 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:41.688 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:42.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:43.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:43.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:43.781 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:08:43.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:08:43.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:43.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:45.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:46.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:46.224 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:46.225 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:08:46.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:08:46.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:46.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:47.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:48.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:48.240 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:48.240 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:08:48.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:08:48.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:48.241 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:49.107 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:50.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:50.109 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:50.109 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:08:50.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:08:50.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:50.110 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:51.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:52.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:52.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:52.439 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:08:52.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:08:52.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:52.440 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:53.519 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:54.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:54.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:54.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:08:54.521 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:08:54.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:54.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:55.141 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:08:56.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:08:56.143 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:08:56.143 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:08:56.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:08:56.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:08:56.144 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:08:56.778 INFO:tasks.workunit.client.0.smithi137.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2337814609,v1:127.0.0.1:6811/2337814609] [v2:127.0.0.1:6812/2337814609,v1:127.0.0.1:6813/2337814609] exists,up 013a7c53-97c7-481e-aa2c-1d43a0d425a7 2024-04-12T17:08:56.781 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:08:56.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:08:56.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:08:56.784 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T17:08:56.785 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:08:56.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37c70af7-3920-434a-a8c7-29b811e67ebf 2024-04-12T17:08:56.785 INFO:tasks.workunit.client.0.smithi137.stdout:add osd2 37c70af7-3920-434a-a8c7-29b811e67ebf 2024-04-12T17:08:56.785 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 37c70af7-3920-434a-a8c7-29b811e67ebf' 2024-04-12T17:08:56.786 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:08:56.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoahlm213iLxAAi3XDzsyoFr+NJ+O07oPbSg== 2024-04-12T17:08:56.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoahlm213iLxAAi3XDzsyoFr+NJ+O07oPbSg=="}' 2024-04-12T17:08:56.805 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37c70af7-3920-434a-a8c7-29b811e67ebf -i td/test-erasure-eio/2/new.json 2024-04-12T17:08:57.459 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:08:57.477 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-04-12T17:08:57.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoahlm213iLxAAi3XDzsyoFr+NJ+O07oPbSg== --osd-uuid 37c70af7-3920-434a-a8c7-29b811e67ebf 2024-04-12T17:08:57.511 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:57.507+0000 7fbcf797e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:57.513 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:57.511+0000 7fbcf797e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:57.514 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:57.511+0000 7fbcf797e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:08:57.515 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:57.511+0000 7fbcf797e7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T17:08:57.515 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:57.511+0000 7fbcf797e7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T17:08:57.515 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:57.511+0000 7fbcf797e7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-04-12T17:08:57.634 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:08:57.631+0000 7fbcf797e7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-04-12T17:09:05.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-04-12T17:09:05.986 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:09:05.987 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd2 key to auth repository 2024-04-12T17:09:05.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-12T17:09:05.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:09:06.553 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.2 2024-04-12T17:09:06.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-12T17:09:06.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:09:06.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:09:06.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:09:06.554 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:09:06.586 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:06.583+0000 7f12f78a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:06.588 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:06.583+0000 7f12f78a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:06.589 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:06.587+0000 7f12f78a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:06.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-12T17:09:06.969 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:09:06.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T17:09:06.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:09:06.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:09:06.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:06.970 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:09:06.970 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:09:06.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:06.971 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:07.139 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:07.135+0000 7f12f78a37c0 -1 Falling back to public interface 2024-04-12T17:09:07.366 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:08.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:08.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:08.368 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:09:08.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:09:08.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:08.369 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:08.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:09.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:09.772 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:09.772 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:09:09.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:09:09.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:09.773 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:10.033 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:10.031+0000 7f12f78a37c0 -1 osd.2 0 log_to_monitors true 2024-04-12T17:09:10.276 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:11.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:11.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:11.278 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:09:11.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:09:11.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:11.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:11.872 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:12.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:12.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:12.874 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:09:12.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:09:12.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:12.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:13.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:14.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:14.470 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:14.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:09:14.471 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:09:14.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:14.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:15.235 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:16.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:16.237 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:16.238 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:09:16.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:09:16.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:16.238 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:16.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:17.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:17.893 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:17.894 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:09:17.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:09:17.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:17.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:18.368 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:19.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:19.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:19.370 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:09:19.370 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:09:19.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:19.371 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:19.847 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:20.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:20.849 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:20.849 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:09:20.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:09:20.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:20.850 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:21.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:22.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:22.419 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:09:22.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:09:22.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:22.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:23.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:24.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:24.118 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:24.118 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:09:24.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:09:24.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:24.119 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:24.569 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:25.570 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:25.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:25.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:09:25.571 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:09:25.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:25.571 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:26.037 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:27.038 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:27.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:27.039 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:09:27.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:09:27.039 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:27.040 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:27.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:28.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:28.680 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:28.680 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:09:28.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:09:28.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:28.681 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:29.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:30.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:30.218 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:30.219 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:09:30.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:09:30.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:30.219 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:30.704 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:31.706 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:31.706 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:09:31.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:09:31.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:31.707 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:32.898 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:33.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:33.900 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:33.900 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:09:33.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:09:33.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:33.901 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:34.433 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:35.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:35.434 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:35.435 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:09:35.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:09:35.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:35.435 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:36.275 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:37.277 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:37.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:37.278 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:09:37.278 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:09:37.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:37.279 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:38.069 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:39.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:39.071 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:39.072 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:09:39.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:09:39.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:39.072 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:40.014 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:41.015 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:41.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:41.016 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:09:41.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:09:41.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:41.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:41.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:42.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:42.830 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:42.831 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T17:09:42.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T17:09:42.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:42.831 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:09:43.545 INFO:tasks.workunit.client.0.smithi137.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/898401817,v1:127.0.0.1:6819/898401817] [v2:127.0.0.1:6820/898401817,v1:127.0.0.1:6821/898401817] exists,up 37c70af7-3920-434a-a8c7-29b811e67ebf 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-12T17:09:43.548 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:09:43.549 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:09:43.550 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:09:43.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:09:43.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:09:43.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-12T17:09:43.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-12T17:09:43.551 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f66dd580-8af7-4a90-9843-681d75f03cc3 2024-04-12T17:09:43.552 INFO:tasks.workunit.client.0.smithi137.stdout:add osd3 f66dd580-8af7-4a90-9843-681d75f03cc3 2024-04-12T17:09:43.553 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f66dd580-8af7-4a90-9843-681d75f03cc3' 2024-04-12T17:09:43.553 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-12T17:09:43.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXahlmoRcQIhAA/4uplE8BMXQUrnlD5dehwA== 2024-04-12T17:09:43.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXahlmoRcQIhAA/4uplE8BMXQUrnlD5dehwA=="}' 2024-04-12T17:09:43.574 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f66dd580-8af7-4a90-9843-681d75f03cc3 -i td/test-erasure-eio/3/new.json 2024-04-12T17:09:44.748 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:09:44.767 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-04-12T17:09:44.769 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXahlmoRcQIhAA/4uplE8BMXQUrnlD5dehwA== --osd-uuid f66dd580-8af7-4a90-9843-681d75f03cc3 2024-04-12T17:09:44.804 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:44.799+0000 7fc04966d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:44.806 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:44.803+0000 7fc04966d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:44.807 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:44.803+0000 7fc04966d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:44.808 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:44.803+0000 7fc04966d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T17:09:44.808 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:44.803+0000 7fc04966d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T17:09:44.808 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:44.803+0000 7fc04966d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-04-12T17:09:44.990 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:44.987+0000 7fc04966d7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-04-12T17:09:53.077 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-04-12T17:09:53.078 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-12T17:09:53.079 INFO:tasks.workunit.client.0.smithi137.stdout:adding osd3 key to auth repository 2024-04-12T17:09:53.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-12T17:09:53.079 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-12T17:09:53.692 INFO:tasks.workunit.client.0.smithi137.stdout:start osd.3 2024-04-12T17:09:53.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-12T17:09:53.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:09:53.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-12T17:09:53.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-12T17:09:53.693 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-12T17:09:53.725 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:53.723+0000 7f68437507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:53.757 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:53.755+0000 7f68437507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:53.758 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:53.755+0000 7f68437507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:09:54.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-12T17:09:54.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:09:54.188 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-12T17:09:54.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:09:54.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:09:54.189 INFO:tasks.workunit.client.0.smithi137.stdout:0 2024-04-12T17:09:54.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:54.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:09:54.189 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:09:54.190 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:54.338 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:54.335+0000 7f68437507c0 -1 Falling back to public interface 2024-04-12T17:09:54.586 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:55.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:55.588 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:55.589 INFO:tasks.workunit.client.0.smithi137.stdout:1 2024-04-12T17:09:55.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:09:55.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:55.589 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:09:56.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:57.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:57.019 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:57.020 INFO:tasks.workunit.client.0.smithi137.stdout:2 2024-04-12T17:09:57.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:09:57.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:57.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:09:57.174 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:09:57.171+0000 7f68437507c0 -1 osd.3 0 log_to_monitors true 2024-04-12T17:09:57.490 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:09:58.491 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:09:58.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:09:58.492 INFO:tasks.workunit.client.0.smithi137.stdout:3 2024-04-12T17:09:58.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:09:58.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:09:58.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:09:59.223 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:00.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:00.225 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:00.225 INFO:tasks.workunit.client.0.smithi137.stdout:4 2024-04-12T17:10:00.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-12T17:10:00.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:00.226 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:00.883 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:01.884 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:01.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:01.885 INFO:tasks.workunit.client.0.smithi137.stdout:5 2024-04-12T17:10:01.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-12T17:10:01.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:01.885 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:02.416 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:03.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:03.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:03.419 INFO:tasks.workunit.client.0.smithi137.stdout:6 2024-04-12T17:10:03.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-12T17:10:03.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:03.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:03.919 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:04.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:04.921 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:04.922 INFO:tasks.workunit.client.0.smithi137.stdout:7 2024-04-12T17:10:04.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-12T17:10:04.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:04.922 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:05.461 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:06.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:06.463 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:06.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-12T17:10:06.464 INFO:tasks.workunit.client.0.smithi137.stdout:8 2024-04-12T17:10:06.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:06.464 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:07.184 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:08.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:08.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:08.186 INFO:tasks.workunit.client.0.smithi137.stdout:9 2024-04-12T17:10:08.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-12T17:10:08.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:08.187 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:08.730 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:09.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:09.732 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:09.732 INFO:tasks.workunit.client.0.smithi137.stdout:10 2024-04-12T17:10:09.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-12T17:10:09.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:09.733 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:10.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:11.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:11.247 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:11.247 INFO:tasks.workunit.client.0.smithi137.stdout:11 2024-04-12T17:10:11.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-12T17:10:11.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:11.248 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:11.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:12.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:12.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:12.756 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-12T17:10:12.757 INFO:tasks.workunit.client.0.smithi137.stdout:12 2024-04-12T17:10:12.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:12.757 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:13.469 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:14.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:14.471 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:14.471 INFO:tasks.workunit.client.0.smithi137.stdout:13 2024-04-12T17:10:14.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-12T17:10:14.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:14.472 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:15.018 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:16.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:16.020 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:16.020 INFO:tasks.workunit.client.0.smithi137.stdout:14 2024-04-12T17:10:16.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-12T17:10:16.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:16.021 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:16.518 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:17.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:17.520 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:17.521 INFO:tasks.workunit.client.0.smithi137.stdout:15 2024-04-12T17:10:17.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-12T17:10:17.521 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:17.522 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:18.437 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:19.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:19.438 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:19.438 INFO:tasks.workunit.client.0.smithi137.stdout:16 2024-04-12T17:10:19.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-12T17:10:19.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:19.439 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:20.325 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:21.327 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-12T17:10:21.327 INFO:tasks.workunit.client.0.smithi137.stdout:17 2024-04-12T17:10:21.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:21.328 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:22.483 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:23.484 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:23.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:23.485 INFO:tasks.workunit.client.0.smithi137.stdout:18 2024-04-12T17:10:23.485 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-12T17:10:23.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:23.486 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:24.559 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:25.561 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:25.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:25.562 INFO:tasks.workunit.client.0.smithi137.stdout:19 2024-04-12T17:10:25.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-12T17:10:25.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:25.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:26.242 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:27.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:27.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:27.244 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-12T17:10:27.245 INFO:tasks.workunit.client.0.smithi137.stdout:20 2024-04-12T17:10:27.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:27.245 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:28.049 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:29.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:29.051 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:29.051 INFO:tasks.workunit.client.0.smithi137.stdout:21 2024-04-12T17:10:29.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-12T17:10:29.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:29.052 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:29.826 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:30.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:30.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:30.829 INFO:tasks.workunit.client.0.smithi137.stdout:22 2024-04-12T17:10:30.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-12T17:10:30.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:30.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-12T17:10:31.303 INFO:tasks.workunit.client.0.smithi137.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2325325255,v1:127.0.0.1:6827/2325325255] [v2:127.0.0.1:6828/2325325255,v1:127.0.0.1:6829/2325325255] exists,up f66dd580-8af7-4a90-9843-681d75f03cc3 2024-04-12T17:10:31.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:10:31.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:10:31.304 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:10:31.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-04-12T17:10:31.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-12T17:10:31.305 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-12T17:10:31.306 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-12T17:10:31.306 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:10:31.306 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:10:31.306 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79474/ceph-osd.0.asok 2024-04-12T17:10:31.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-04-12T17:10:31.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79474/ceph-osd.0.asok log flush 2024-04-12T17:10:31.403 INFO:tasks.workunit.client.0.smithi137.stdout:{} 2024-04-12T17:10:31.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-04-12T17:10:31.419 INFO:tasks.workunit.client.0.smithi137.stdout:2024-04-12T17:07:34.267+0000 7f1c354ed7c0 0 load: jerasure load: lrc 2024-04-12T17:10:31.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-04-12T17:10:31.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:320: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-04-12T17:10:31.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-04-12T17:10:31.420 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-04-12T17:10:31.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-04-12T17:10:31.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-04-12T17:10:31.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-04-12T17:10:31.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-04-12T17:10:31.421 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-04-12T17:10:32.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-04-12T17:10:32.738 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-04-12T17:10:34.200 INFO:tasks.workunit.client.0.smithi137.stderr:pool 'pool-jerasure' already exists 2024-04-12T17:10:34.217 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-12T17:10:35.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-04-12T17:10:35.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:10:35.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:10:35.220 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:10:35.220 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:10:35.222 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:10:35.222 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:10:35.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:10:35.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:10:35.222 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:10:35.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:10:35.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:10:35.346 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:10:35.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:10:35.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:10:35.347 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:10:35.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:10:35.920 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:10:35.920 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:10:35.920 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:10:35.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:10:35.920 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:10:35.920 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:10:36.136 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803815 2024-04-12T17:10:36.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803815 2024-04-12T17:10:36.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803815' 2024-04-12T17:10:36.137 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:10:36.137 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:10:36.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574874 2024-04-12T17:10:36.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574874 2024-04-12T17:10:36.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803815 1-55834574874' 2024-04-12T17:10:36.298 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:10:36.298 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:10:36.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378645 2024-04-12T17:10:36.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378645 2024-04-12T17:10:36.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803815 1-55834574874 2-81604378645' 2024-04-12T17:10:36.454 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:10:36.455 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:10:36.605 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117001 2024-04-12T17:10:36.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117001 2024-04-12T17:10:36.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803815 1-55834574874 2-81604378645 3-115964117001' 2024-04-12T17:10:36.606 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:10:36.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803815 2024-04-12T17:10:36.606 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:10:36.607 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:10:36.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803815 2024-04-12T17:10:36.608 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:10:36.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803815 2024-04-12T17:10:36.609 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-04-12T17:10:36.610 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.0 seq 25769803815 2024-04-12T17:10:36.610 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:10:37.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803813 -lt 25769803815 2024-04-12T17:10:37.022 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:10:38.024 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:10:38.025 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:10:38.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803815 -lt 25769803815 2024-04-12T17:10:38.476 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:10:38.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574874 2024-04-12T17:10:38.477 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:10:38.479 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:10:38.480 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574874 2024-04-12T17:10:38.480 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:10:38.481 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574874 2024-04-12T17:10:38.482 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-04-12T17:10:38.482 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.1 seq 55834574874 2024-04-12T17:10:38.482 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:10:38.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574874 -lt 55834574874 2024-04-12T17:10:38.891 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:10:38.892 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378645 2024-04-12T17:10:38.892 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:10:38.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:10:38.894 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378645 2024-04-12T17:10:38.894 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:10:38.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378645 2024-04-12T17:10:38.896 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378645' 2024-04-12T17:10:38.896 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.2 seq 81604378645 2024-04-12T17:10:38.896 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:10:39.347 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378645 -lt 81604378645 2024-04-12T17:10:39.348 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:10:39.348 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117001 2024-04-12T17:10:39.349 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:10:39.350 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:10:39.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117001 2024-04-12T17:10:39.351 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:10:39.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117001 2024-04-12T17:10:39.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117001' 2024-04-12T17:10:39.353 INFO:tasks.workunit.client.0.smithi137.stdout:waiting osd.3 seq 115964117001 2024-04-12T17:10:39.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:10:39.814 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117001 -lt 115964117001 2024-04-12T17:10:39.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:10:39.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:10:39.815 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:10:40.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:10:40.359 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:10:40.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:10:40.360 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:10:40.361 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:10:40.361 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:10:40.361 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:10:40.361 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:10:40.770 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:10:40.771 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:10:40.771 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:10:40.772 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:10:41.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:10:41.305 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:10:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:10:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-04-12T17:10:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-04-12T17:10:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-04-12T17:10:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-04-12T17:10:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-04-12T17:10:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-04-12T17:10:41.306 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79474-1-10 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79474-1-10 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79474-1-10 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-04-12T17:10:41.307 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:10:41.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-04-12T17:10:41.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-12T17:10:41.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-04-12T17:10:41.308 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79474-1-10 td/test-erasure-eio/ORIGINAL 2024-04-12T17:10:41.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79474-1-10 td/test-erasure-eio 1 10 set 2024-04-12T17:10:41.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79474-1-10 2024-04-12T17:10:41.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-04-12T17:10:41.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-04-12T17:10:41.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-04-12T17:10:41.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-04-12T17:10:41.562 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-04-12T17:10:41.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-04-12T17:10:41.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-04-12T17:10:41.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-04-12T17:10:41.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-04-12T17:10:41.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79474-1-10 2024-04-12T17:10:41.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:10:41.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79474-1-10 2024-04-12T17:10:41.563 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79474-1-10 2024-04-12T17:10:41.564 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:10:42.008 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:10:42.008 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:10:42.008 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T17:10:42.009 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-04-12T17:10:42.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-04-12T17:10:42.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-04-12T17:10:42.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-04-12T17:10:42.009 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-04-12T17:10:42.851 INFO:tasks.workunit.client.0.smithi137.stderr:noout is set 2024-04-12T17:10:42.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-04-12T17:10:42.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-04-12T17:10:42.871 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-04-12T17:10:42.874 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-04-12T17:10:42.874 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-04-12T17:10:42.874 INFO:tasks.workunit.client.0.smithi137.stderr:10 bytes copied, 0.00020334 s, 49.2 kB/s 2024-04-12T17:10:42.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79474-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:10:42.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:10:42.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:10:42.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-12T17:10:42.874 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79474-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:10:42.875 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:10:44.185 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:10:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79474-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:10:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:10:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:10:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-12T17:10:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:10:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-04-12T17:10:44.186 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79474-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:10:47.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-04-12T17:10:47.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:10:47.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:10:47.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-12T17:10:47.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:10:47.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-04-12T17:10:47.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:10:47.080 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:10:47.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:10:47.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:10:47.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-04-12T17:10:47.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-04-12T17:10:47.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:10:47.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:10:47.081 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:10:47.083 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:10:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:10:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:10:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:10:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:10:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:10:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:10:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-04-12T17:10:47.084 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-12T17:10:47.085 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.1 2024-04-12T17:10:47.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:10:47.085 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-04-12T17:10:47.085 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-12T17:10:47.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:10:47.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:10:47.087 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:10:47.118 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:10:47.114+0000 7f7cec7847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:10:47.121 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:10:47.118+0000 7f7cec7847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:10:47.122 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:10:47.118+0000 7f7cec7847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:10:47.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-12T17:10:47.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:10:47.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-12T17:10:47.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:10:47.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:10:47.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:47.510 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:10:47.511 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:10:47.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:47.511 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:10:47.678 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:10:47.674+0000 7f7cec7847c0 -1 Falling back to public interface 2024-04-12T17:10:47.943 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:48.944 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:48.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:48.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:10:48.945 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:10:48.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:48.945 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:10:49.353 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:50.354 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:50.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:50.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:10:50.355 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:10:50.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:50.355 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:10:50.667 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:10:50.662+0000 7f7cec7847c0 -1 osd.1 36 log_to_monitors true 2024-04-12T17:10:50.780 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:10:51.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:10:51.782 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:10:51.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:10:51.783 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:10:51.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:10:51.783 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-12T17:10:52.397 INFO:tasks.workunit.client.0.smithi137.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 38 last_clean_interval [13,36) [v2:127.0.0.1:6810/2781997199,v1:127.0.0.1:6811/2781997199] [v2:127.0.0.1:6812/2781997199,v1:127.0.0.1:6813/2781997199] exists,up 013a7c53-97c7-481e-aa2c-1d43a0d425a7 2024-04-12T17:10:52.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:10:52.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:10:52.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:10:52.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:10:52.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:10:52.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:10:52.397 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:10:52.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:10:52.398 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:10:52.398 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:10:52.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:10:52.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:10:52.398 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:10:52.505 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:10:52.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:10:52.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:10:52.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:10:52.506 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:10:52.507 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:10:53.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:10:53.208 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:10:53.208 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:10:53.208 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:10:53.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:10:53.208 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:10:53.208 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:10:53.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803820 2024-04-12T17:10:53.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803820 2024-04-12T17:10:53.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820' 2024-04-12T17:10:53.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:10:53.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:10:53.616 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626435 2024-04-12T17:10:53.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626435 2024-04-12T17:10:53.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-180388626435' 2024-04-12T17:10:53.617 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:10:53.617 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:10:54.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378651 2024-04-12T17:10:54.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378651 2024-04-12T17:10:54.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-180388626435 2-81604378651' 2024-04-12T17:10:54.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:10:54.007 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:10:54.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117006 2024-04-12T17:10:54.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117006 2024-04-12T17:10:54.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-180388626435 2-81604378651 3-115964117006' 2024-04-12T17:10:54.166 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:10:54.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803820 2024-04-12T17:10:54.167 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:10:54.167 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:10:54.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803820 2024-04-12T17:10:54.168 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:10:54.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803820 2024-04-12T17:10:54.169 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-04-12T17:10:54.169 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803820 2024-04-12T17:10:54.170 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:10:54.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803820 -lt 25769803820 2024-04-12T17:10:54.560 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:10:54.561 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-180388626435 2024-04-12T17:10:54.561 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:10:54.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:10:54.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-180388626435 2024-04-12T17:10:54.563 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:10:54.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626435 2024-04-12T17:10:54.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 180388626435' 2024-04-12T17:10:54.565 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 180388626435 2024-04-12T17:10:54.566 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:10:54.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 180388626435 2024-04-12T17:10:54.964 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:10:55.966 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:10:55.966 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:10:56.418 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626435 -lt 180388626435 2024-04-12T17:10:56.419 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:10:56.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378651 2024-04-12T17:10:56.420 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:10:56.422 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:10:56.423 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378651 2024-04-12T17:10:56.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:10:56.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378651 2024-04-12T17:10:56.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378651' 2024-04-12T17:10:56.424 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 81604378651 2024-04-12T17:10:56.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:10:56.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378651 -lt 81604378651 2024-04-12T17:10:56.864 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:10:56.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117006 2024-04-12T17:10:56.865 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:10:56.867 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:10:56.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117006 2024-04-12T17:10:56.868 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:10:56.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117006 2024-04-12T17:10:56.870 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117006' 2024-04-12T17:10:56.870 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 115964117006 2024-04-12T17:10:56.871 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:10:57.352 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117006 -lt 115964117006 2024-04-12T17:10:57.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:10:57.353 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:10:57.354 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:10:57.915 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:10:57.916 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:10:57.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:10:57.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:10:57.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:10:57.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:10:57.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:10:57.916 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:10:58.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:10:58.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:10:58.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:10:58.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:10:58.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:10:58.894 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:10:58.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:10:58.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-04-12T17:10:58.895 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-04-12T17:10:59.645 INFO:tasks.workunit.client.0.smithi137.stderr:noout is unset 2024-04-12T17:10:59.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79474-1-10 2024-04-12T17:10:59.667 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:10:59.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:10:59.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79474-1-10 2024-04-12T17:10:59.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-04-12T17:10:59.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-04-12T17:10:59.668 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79474-1-10 td/test-erasure-eio/COPY 2024-04-12T17:10:59.979 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-04-12T17:10:59.980 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-04-12T17:10:59.982 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-04-12T17:10:59.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-04-12T17:10:59.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79474-1-10 td/test-erasure-eio 2 10 set 2024-04-12T17:10:59.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79474-1-10 2024-04-12T17:10:59.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-04-12T17:10:59.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-04-12T17:10:59.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-04-12T17:10:59.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-04-12T17:10:59.984 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-04-12T17:10:59.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-04-12T17:10:59.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-04-12T17:10:59.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-04-12T17:10:59.985 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-04-12T17:10:59.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79474-1-10 2024-04-12T17:10:59.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-04-12T17:10:59.986 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79474-1-10 2024-04-12T17:10:59.986 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79474-1-10 2024-04-12T17:10:59.986 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-12T17:11:00.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-12T17:11:00.409 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:11:00.409 INFO:tasks.workunit.client.0.smithi137.stderr:2' 2024-04-12T17:11:00.409 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-04-12T17:11:00.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-04-12T17:11:00.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-04-12T17:11:00.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-04-12T17:11:00.409 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-04-12T17:11:01.501 INFO:tasks.workunit.client.0.smithi137.stderr:noout is set 2024-04-12T17:11:01.523 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-04-12T17:11:01.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-04-12T17:11:01.524 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-04-12T17:11:01.526 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records in 2024-04-12T17:11:01.526 INFO:tasks.workunit.client.0.smithi137.stderr:1+0 records out 2024-04-12T17:11:01.526 INFO:tasks.workunit.client.0.smithi137.stderr:10 bytes copied, 0.000175214 s, 57.1 kB/s 2024-04-12T17:11:01.526 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79474-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79474-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:11:01.527 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:11:01.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:11:01.528 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:11:02.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:11:02.840 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79474-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:11:02.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-04-12T17:11:02.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-12T17:11:02.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-12T17:11:02.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-12T17:11:02.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-04-12T17:11:02.841 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79474-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-04-12T17:11:05.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-04-12T17:11:05.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-04-12T17:11:05.673 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-12T17:11:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-12T17:11:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-12T17:11:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-04-12T17:11:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-04-12T17:11:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-12T17:11:05.674 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:11:05.675 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-12T17:11:05.677 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-12T17:11:05.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-04-12T17:11:05.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-12T17:11:05.678 INFO:tasks.workunit.client.0.smithi137.stderr:start osd.2 2024-04-12T17:11:05.678 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d201fb60-24f7-44c5-96ba-019d11a04ee5 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79474/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-12T17:11:05.678 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-04-12T17:11:05.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-12T17:11:05.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-12T17:11:05.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-12T17:11:05.679 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-12T17:11:05.711 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:11:05.706+0000 7fd077b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:11:05.713 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:11:05.710+0000 7fd077b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:11:05.715 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:11:05.710+0000 7fd077b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-12T17:11:06.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-12T17:11:06.115 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-12T17:11:06.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-12T17:11:06.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-12T17:11:06.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-12T17:11:06.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:11:06.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-12T17:11:06.116 INFO:tasks.workunit.client.0.smithi137.stderr:0 2024-04-12T17:11:06.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:11:06.116 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:11:06.262 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:11:06.258+0000 7fd077b1d7c0 -1 Falling back to public interface 2024-04-12T17:11:06.563 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:11:07.565 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:11:07.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:11:07.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-12T17:11:07.566 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:11:07.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:11:07.566 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:11:08.004 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:11:09.005 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:11:09.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:11:09.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-12T17:11:09.006 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:11:09.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:11:09.006 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:11:09.162 INFO:tasks.workunit.client.0.smithi137.stderr:2024-04-12T17:11:09.158+0000 7fd077b1d7c0 -1 osd.2 46 log_to_monitors true 2024-04-12T17:11:09.492 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-12T17:11:10.493 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-12T17:11:10.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-12T17:11:10.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-12T17:11:10.494 INFO:tasks.workunit.client.0.smithi137.stderr:3 2024-04-12T17:11:10.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-12T17:11:10.494 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-12T17:11:10.977 INFO:tasks.workunit.client.0.smithi137.stderr:osd.2 up in weight 1 up_from 53 up_thru 38 down_at 49 last_clean_interval [19,46) [v2:127.0.0.1:6818/3568536262,v1:127.0.0.1:6819/3568536262] [v2:127.0.0.1:6820/3568536262,v1:127.0.0.1:6821/3568536262] exists,up 37c70af7-3920-434a-a8c7-29b811e67ebf 2024-04-12T17:11:10.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-12T17:11:10.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-12T17:11:10.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-12T17:11:10.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-12T17:11:10.977 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-12T17:11:10.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-12T17:11:10.978 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-12T17:11:10.978 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-12T17:11:10.978 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-12T17:11:10.979 INFO:tasks.workunit.client.0.smithi137.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-12T17:11:10.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-12T17:11:10.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-12T17:11:10.979 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-12T17:11:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-12T17:11:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-12T17:11:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-12T17:11:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-12T17:11:11.086 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-12T17:11:11.086 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-12T17:11:11.659 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-12T17:11:11.660 INFO:tasks.workunit.client.0.smithi137.stderr:1 2024-04-12T17:11:11.660 INFO:tasks.workunit.client.0.smithi137.stderr:2 2024-04-12T17:11:11.660 INFO:tasks.workunit.client.0.smithi137.stderr:3' 2024-04-12T17:11:11.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-12T17:11:11.660 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:11:11.660 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-12T17:11:11.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803826 2024-04-12T17:11:11.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803826 2024-04-12T17:11:11.828 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826' 2024-04-12T17:11:11.829 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:11:11.829 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-12T17:11:11.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626441 2024-04-12T17:11:11.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626441 2024-04-12T17:11:11.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-180388626441' 2024-04-12T17:11:11.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:11:11.988 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-12T17:11:12.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266691 2024-04-12T17:11:12.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266691 2024-04-12T17:11:12.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-180388626441 2-227633266691' 2024-04-12T17:11:12.202 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-12T17:11:12.202 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-12T17:11:12.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117012 2024-04-12T17:11:12.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117012 2024-04-12T17:11:12.361 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-180388626441 2-227633266691 3-115964117012' 2024-04-12T17:11:12.362 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:11:12.362 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803826 2024-04-12T17:11:12.362 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:11:12.363 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-12T17:11:12.363 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803826 2024-04-12T17:11:12.364 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:11:12.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803826 2024-04-12T17:11:12.365 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-04-12T17:11:12.365 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.0 seq 25769803826 2024-04-12T17:11:12.365 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:11:12.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803824 -lt 25769803826 2024-04-12T17:11:12.771 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-12T17:11:13.777 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-12T17:11:13.945 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-12T17:11:14.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803826 -lt 25769803826 2024-04-12T17:11:14.230 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:11:14.231 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-180388626441 2024-04-12T17:11:14.231 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:11:14.232 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-12T17:11:14.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-180388626441 2024-04-12T17:11:14.233 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:11:14.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626441 2024-04-12T17:11:14.234 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 180388626441' 2024-04-12T17:11:14.234 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.1 seq 180388626441 2024-04-12T17:11:14.234 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-12T17:11:14.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626441 -lt 180388626441 2024-04-12T17:11:14.632 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:11:14.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-227633266691 2024-04-12T17:11:14.632 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:11:14.634 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-12T17:11:14.634 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-227633266691 2024-04-12T17:11:14.635 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:11:14.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266691 2024-04-12T17:11:14.636 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 227633266691' 2024-04-12T17:11:14.636 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.2 seq 227633266691 2024-04-12T17:11:14.636 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-12T17:11:15.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266691 -lt 227633266691 2024-04-12T17:11:15.061 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-12T17:11:15.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117012 2024-04-12T17:11:15.062 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-12T17:11:15.063 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-12T17:11:15.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117012 2024-04-12T17:11:15.064 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-12T17:11:15.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117012 2024-04-12T17:11:15.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117012' 2024-04-12T17:11:15.066 INFO:tasks.workunit.client.0.smithi137.stderr:waiting osd.3 seq 115964117012 2024-04-12T17:11:15.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-12T17:11:15.460 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117012 -lt 115964117012 2024-04-12T17:11:15.460 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-12T17:11:15.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:11:15.461 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:11:16.016 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-12T17:11:16.017 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-12T17:11:16.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-12T17:11:16.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-12T17:11:16.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-12T17:11:16.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-12T17:11:16.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-12T17:11:16.017 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-12T17:11:16.424 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-12T17:11:16.424 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-12T17:11:16.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-12T17:11:16.425 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-12T17:11:16.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-12T17:11:16.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-12T17:11:16.991 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-12T17:11:16.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-04-12T17:11:16.992 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-04-12T17:11:17.857 INFO:tasks.workunit.client.0.smithi137.stderr:noout is unset 2024-04-12T17:11:17.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79474-1-10 fail 2024-04-12T17:11:17.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-04-12T17:11:17.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-04-12T17:11:17.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79474-1-10 2024-04-12T17:11:17.879 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-04-12T17:11:17.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-04-12T17:11:17.880 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79474-1-10 td/test-erasure-eio/COPY 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: return 1 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: return 1 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: return 1 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-eio 1 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-04-12T17:11:18.065 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-04-12T17:11:18.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-12T17:11:18.066 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-12T17:11:18.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-12T17:11:18.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-12T17:11:18.066 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-12T17:11:18.382 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-12T17:11:18.383 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-12T17:11:18.384 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-12T17:11:18.385 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-12T17:11:18.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-12T17:11:18.386 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-12T17:11:18.387 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-12T17:11:18.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:11:18.388 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-12T17:11:18.389 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-12T17:11:18.389 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T17:11:18.390 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-12T17:11:18.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-04-12T17:11:18.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-04-12T17:11:18.392 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-04-12T17:11:18.393 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/test-erasure-eio/mgr.x.log td/test-erasure-eio/mon.a.log td/test-erasure-eio/osd.0.log td/test-erasure-eio/osd.1.log td/test-erasure-eio/osd.2.log td/test-erasure-eio/osd.3.log /home/ubuntu/cephtest/archive/log 2024-04-12T17:11:18.395 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-04-12T17:11:18.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-12T17:11:18.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-12T17:11:18.413 INFO:tasks.workunit.client.0.smithi137.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79474 2024-04-12T17:11:18.413 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79474 2024-04-12T17:11:18.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-12T17:11:18.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-12T17:11:18.414 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-04-12T17:11:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T17:11:18.416 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-04-12T17:11:18.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-12T17:11:18.765 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf9c5618cc018927fa77780d25b6deb1f5dc254d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh' 2024-04-12T17:11:18.994 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b28c5c38df2945429b3a64d1ed53af88 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf9c5618cc018927fa77780d25b6deb1f5dc254d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh' 2024-04-12T17:11:18.997 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-12T17:11:19.006 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-12T17:11:19.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T17:11:19.049 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-12T17:11:19.058 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-12T17:11:19.058 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T17:11:19.081 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2024-04-12T17:11:19.081 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2024-04-12T17:11:19.082 INFO:teuthology.orchestra.run.smithi137.stdout:*hv01.front.sepi 67.205.162.81 3 u 18 64 377 0.092 -0.716 0.361 2024-04-12T17:11:19.082 INFO:teuthology.orchestra.run.smithi137.stdout:+hv02.front.sepi 63.231.80.2 3 u 29 64 377 0.099 +1.912 0.285 2024-04-12T17:11:19.082 INFO:teuthology.orchestra.run.smithi137.stdout:+hv03.front.sepi 74.6.168.72 3 u 3 64 377 0.101 +0.357 0.575 2024-04-12T17:11:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-12T17:11:19.082 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-12T17:11:19.095 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-12T17:11:19.096 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-12T17:11:19.105 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-12T17:11:19.112 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-12T17:11:19.120 INFO:teuthology.task.internal:Duration was 4829.643571 seconds 2024-04-12T17:11:19.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-12T17:11:19.129 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-12T17:11:19.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T17:11:19.170 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-12T17:11:19.170 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-04-12T17:11:19.171 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T17:11:19.222 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-12T17:11:19.222 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T17:11:19.318 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-12T17:11:19.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T17:11:19.370 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-12T17:11:19.382 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-12T17:11:19.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T17:11:19.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-12T17:11:19.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T17:11:19.445 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-04-12T17:11:19.477 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T17:11:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T17:11:19.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-12T17:11:19.535 INFO:teuthology.task.internal:Transferring archived files... 2024-04-12T17:11:19.537 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654145/remote/smithi137 2024-04-12T17:11:19.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T17:11:20.790 INFO:teuthology.task.internal:Removing archive directory... 2024-04-12T17:11:20.791 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T17:11:20.834 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-12T17:11:20.844 INFO:teuthology.task.internal:Not uploading archives. 2024-04-12T17:11:20.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-12T17:11:20.853 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-12T17:11:20.853 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T17:11:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: 658640 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 12 17:11 /home/ubuntu/cephtest 2024-04-12T17:11:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: 788842 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Apr 12 15:59 /home/ubuntu/cephtest/mnt.0 2024-04-12T17:11:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: 790638 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Apr 12 16:19 /home/ubuntu/cephtest/mnt.0/client.0 2024-04-12T17:11:20.882 INFO:teuthology.orchestra.run.smithi137.stdout: 793478 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Apr 12 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-12T17:11:20.883 INFO:teuthology.orchestra.run.smithi137.stdout: 807931 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Apr 12 17:11 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-04-12T17:11:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T17:11:20.884 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-12T17:11:20.884 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-12T17:11:20.889 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-12T17:11:20.900 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-12T17:11:20.930 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi137 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-12T17:11:20.930 INFO:teuthology.nuke:Checking targets against current locks 2024-04-12T17:11:20.954 DEBUG:teuthology.nuke:shortname: smithi137 2024-04-12T17:11:20.954 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T17:11:20.977 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654145', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 15:42:36.932597', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHcJ/LglOklgXMH88njrkk534WEdOIsHeRT1ocPU7vO+2DB6/3zevQgvjesn20Y2jNx4T/3Bya+JOf8h0U52kA='} 2024-04-12T17:11:21.027 INFO:teuthology.orchestra.console.smithi137:Power off 2024-04-12T17:11:21.027 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T17:11:21.051 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-04-12T17:11:21.051 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T17:11:21.075 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-04-12T17:11:25.077 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T17:11:25.188 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-04-12T17:11:29.190 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T17:11:29.301 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-04-12T17:11:29.402 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-04-12T17:11:29.508 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} duration: 4829.6435713768005 failure_reason: 'Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi137 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bf9c5618cc018927fa77780d25b6deb1f5dc254d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh''' flavor: default owner: scheduled_rzarzynski@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b28c5c38df2945429b3a64d1ed53af88 status: fail success: false 2024-04-12T17:11:29.509 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T17:11:29.604 INFO:teuthology.run:FAIL